[GitHub] spark issue #19983: [SPARK-22788][streaming] Use correct hadoop config for f...

2017-12-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19983 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19983: [SPARK-22788][streaming] Use correct hadoop config for f...

2017-12-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19983 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158200059 --- Diff: launcher/src/main/java/org/apache/spark/launcher/ChildProcAppHandle.java --- @@ -171,10 +90,11 @@ void monitorChild

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158200163 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158201743 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -210,8 +209,13 @@ int getPort() { * Removes the client handle

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158201527 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -260,24 +264,30 @@ private long getConnectionTimeout

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158200240 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158198274 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158201917 --- Diff: launcher/src/main/java/org/apache/spark/launcher/package-info.java --- @@ -49,6 +49,15 @@ * * * + * Applications can also

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158307838 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -210,8 +209,13 @@ int getPort() { * Removes the client handle

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158316971 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -210,8 +209,13 @@ int getPort() { * Removes the client handle

[GitHub] spark pull request #19681: [SPARK-20652][sql] Store SQL UI data in the new a...

2017-12-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19681#discussion_r158326122 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala --- @@ -0,0 +1,366 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20047: [SPARK-22861][SQL] SQLAppStatusListener handles m...

2017-12-21 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20047 [SPARK-22861][SQL] SQLAppStatusListener handles multi-job executions. When one execution has multiple jobs, we need to append to the set of stages, not replace them on every job. Added

[GitHub] spark issue #20047: [SPARK-22861][SQL] SQLAppStatusListener handles multi-jo...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20047 @vanzin @carsonwang --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17619 Sorry I am only just looking at this now -- I am not so sure this is doing what you think. I think the notion of "task" in MesosCoarseGrainedSchedulerBackend might be something

[GitHub] spark issue #20045: [Spark-22360][SQL] Add unit tests for Window Specificati...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20045 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20046 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19591 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 this change looks fine -- but I'm having trouble connecting this to the original bug. I guess really I'm confused about how driver logs ever worked ... where is the SparkListenerExecutorA

[GitHub] spark issue #20045: [Spark-22360][SQL] Add unit tests for Window Specificati...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20045 failure seems to be unrelated ... Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 that explains how it gets into the list of executors in the UI -- but where does it get the link for the logs? --- - To

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 a, got it. sorry I misunderstood the comments on the jira ... I thought your second comment meant the original description was incorrect. anyway -- lgtm

[GitHub] spark issue #20038: [SPARK-22836][ui] Show driver logs in UI when available.

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20038 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158509926 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158514239 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158390726 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158516782 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158513454 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158521861 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158515991 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158510545 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158517432 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158523678 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158560723 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158568388 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala --- @@ -0,0 +1,310 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2017-12-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17619 ok I think I understand. This sounds like the equivalent of some of the existing blacklisting behavior which current only exists on yarn -- when a request is made to yarn, the spark context tells

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19848 +1 to Marcelo's comment about having this conversation somewhere archived. I actually feel like this is something hadoop should be documenting ... we are talking about how committe

[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2017-12-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17619 @andreimaximov that is still sort of the case for all cluster managers. You shouldn't get starvation, you should see the app actively fail (SPARK-15865 was the main change, though some small f

[GitHub] spark issue #20038: [SPARK-22836][ui] Show driver logs in UI when available.

2017-12-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20038 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19591 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assignin...

2017-12-28 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20107 [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira on Merge In general jiras are assigned to the original reporter or one of the commentors. This updates the merge script to give you a

[GitHub] spark issue #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira ...

2017-12-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20107 I've just done some light testing on this so far -- lemme test it some more before we merge it. But thought I'd let folks try it out since it seems to be work

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20039#discussion_r159074536 --- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala --- @@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20039#discussion_r159075179 --- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala --- @@ -125,13 +128,39 @@ private[spark] class LiveListenerBus(conf

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20039#discussion_r159080455 --- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala --- @@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf

[GitHub] spark issue #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira ...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20107 whoops this is actually broken -- I didn't push exactly what I had locally :( will send a follow up. --- - To unsubscri

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20011 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira ...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20107 https://github.com/apache/spark/pull/20117 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assi...

2017-12-29 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20117 [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning Small bug fix from last pr, ran a successful merge with this code. You can merge this pull request into a Git repository by running

[GitHub] spark issue #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20117 @srowen @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #20050: [SPARK-22864][core] Disable allocation schedule in Execu...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20050 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19992: [SPARK-22805][CORE] Use StorageLevel aliases in e...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19992#discussion_r159083551 --- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala --- @@ -2022,12 +1947,7 @@ private[spark] object JsonProtocolSuite extends

[GitHub] spark issue #18801: SPARK-10878 Fix race condition when multiple clients res...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18801 +1 on using UUID and having a better test case --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20090: [SPARK-22907]Clean broadcast garbage when IOException oc...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20090 Thanks for submitting this @liupc. The jira description makes sense, though I need to take a closer look at the code to confirm things. But at least two things need to be fixed before this can be

[GitHub] spark issue #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20117 I'm going to merge this since I've tested it out, and the merge script is worse than before until this is in ... --- --

[GitHub] spark issue #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20117 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 merged to master / 2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19848 @steveloughran can you bring this up on dev@? we should move this discussion off of this PR. (sorry haven't had a chance to look yet, but I appreciate you doing

[GitHub] spark pull request #19992: [SPARK-22805][CORE] Use StorageLevel aliases in e...

2018-01-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19992#discussion_r159812291 --- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala --- @@ -2022,12 +1947,7 @@ private[spark] object JsonProtocolSuite extends

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19992 change is fine, but from discussion on the jira I'm unclear if this is really worth it -- gain seems pretty small after the other fix i

[GitHub] spark issue #20082: [SPARK-22897][CORE]: Expose stageAttemptId in TaskContex...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20082 sorry this is a little late, but lgtm too. agree with the points above about leaving the old name deprecated and moving to the new name

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159992789 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore( if

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159990658 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore( if

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159988995 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore( if

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r160017859 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -119,118 +121,115 @@ private class LiveTask( import

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20013 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20056: [SPARK-22878] [CORE] Count totalDroppedEvents for LiveLi...

2018-01-10 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20056 I have the same question as @jiangxb1987 , what is the situation where you'd use this metric? the jira doesn't say either. seems like existing metrics mostly

[GitHub] spark pull request #20236: [SPARK-23044] Error handling for jira assignment

2018-01-11 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20236 [SPARK-23044] Error handling for jira assignment ## What changes were proposed in this pull request? In case the selected user isn't a contributor yet, or any other unexpected error,

[GitHub] spark issue #20236: [SPARK-23044] Error handling for jira assignment

2018-01-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20236 @jerryshao this should fix it, but I don't have anything to merge to test this out -- would appreciate if someone could try it before we

[GitHub] spark issue #20236: [SPARK-23044] Error handling for jira assignment

2018-01-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20236 @vanzin what exactly are you looking for? The one thing which would be easy is letting you write in an arbitrary jira id (no name searching or anything), that sound OK? I guess this bug

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161082423 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -405,49 +404,70 @@ private[history] class FsHistoryProvider

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161095561 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -405,49 +404,70 @@ private[history] class FsHistoryProvider

[GitHub] spark issue #20138: [SPARK-20664][core] Delete stale application data from S...

2018-01-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20138 well, perhaps I mis-represented this -- you still need to turn the event log cleaning on explicitly with the old option, "spark.history.fs.cleaner.enabled". This just doesn

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161098356 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -544,73 +621,75 @@ private[history] class FsHistoryProvider

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161099310 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -834,6 +906,9 @@ private[history] case class

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161099778 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala --- @@ -663,6 +665,95 @@ class FsHistoryProviderSuite extends

[GitHub] spark issue #20056: [SPARK-22878] [CORE] Count totalDroppedEvents for LiveLi...

2018-01-15 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20056 I see that `LiveListenerBus.droppedEventsCounter` and `lastReportTimestamp` are unused, so it certainly makes sense to clean them up one way or the other -- but that might mean we should delete them

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-15 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161660926 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala --- @@ -663,6 +665,95 @@ class FsHistoryProviderSuite extends

[GitHub] spark issue #21104: [SPARK-24021][CORE] fix bug in BlacklistTracker's update...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21104 lgtm, good catch --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21104: [SPARK-24021][CORE] fix bug in BlacklistTracker's update...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21104 merged to master & 2.3.1, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional command

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21068 @tgravescs on the blacklist ratio for task-based blacklisting -- there is nothing, but there are some related jiras: [SPARK-22148](https://issues.apache.org/jira/browse/SPARK-22148) & [SPARK-1

[GitHub] spark issue #20640: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20640 Hi Everyone, just wanted to let you know that SPARK-16630 is progressing now https://github.com/apache/spark/pull/21068 and after some discussion, the implementation will actually need to live

[GitHub] spark pull request #21085: [SPARK-23948] Trigger mapstage's job listener in ...

2018-04-19 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/21085 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21085: [SPARK-23948] Trigger mapstage's job listener in submitM...

2018-04-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21085 whoops, thanks for the reminder @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-23 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/21131 [SPARK-23433][CORE] Late zombie task completions update all tasksets Fetch failure lead to multiple tasksets which are active for a given stage. A late completion from an earlier attempt of the

[GitHub] spark issue #20998: [SPARK-23888][CORE] correct the comment of hasAttemptOnH...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20998 merged to master, thanks @Ngone51 . I also updated the commit msg some before committing, I thought it best to focus on the eventual change, figured it wasn't worth bugging you for another u

[GitHub] spark issue #21096: [SPARK-24011][CORE][WIP] cache rdd's immediate parent Sh...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21096 its not a bad idea, but as @markhamstra mentions we can't have an `rddToImmediateShuffleDependency` data structure which keeps growing. You could keep it local to one job submission, which

[GitHub] spark issue #21132: [SPARK-24029][core] Follow up: set SO_REUSEADDR on the s...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21132 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21131: [SPARK-23433][CORE] Late zombie task completions update ...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21131 @markhamstra @zsxwing @jiangxb1987 @Ngone51 would appreciate a review, thanks --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20940 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183775539 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183776603 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183777930 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -689,6 +689,20 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183778637 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183779968 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183781695 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #20512: [SPARK-23182][CORE] Allow enabling TCP keep alive on the...

2018-04-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20512 this is just far enough outside my expertise I don't have an opinion -- but @zsxwing might have some thoughts --- ---

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-04-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r183793745 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala --- @@ -917,4 +917,111 @@ class TaskSchedulerImplSuite extends

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-04-25 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21068 A couple more high-level thoughts: 1) Do we want to have a event posted about the node getting blacklisted? I think it would be useful. But then there needs to be a msg from the

<    1   2   3   4   5   6   7   8   9   10   >