[GitHub] spark issue #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20117 I'm going to merge this since I've tested it out, and the merge script is worse than before until

[GitHub] spark issue #20090: [SPARK-22907]Clean broadcast garbage when IOException oc...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20090 Thanks for submitting this @liupc. The jira description makes sense, though I need to take a closer look at the code to confirm things. But at least two things need to be fixed before this can

[GitHub] spark issue #18801: SPARK-10878 Fix race condition when multiple clients res...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/18801 +1 on using UUID and having a better test case --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19992: [SPARK-22805][CORE] Use StorageLevel aliases in e...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19992#discussion_r159083551 --- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala --- @@ -2022,12 +1947,7 @@ private[spark] object JsonProtocolSuite extends

[GitHub] spark issue #20050: [SPARK-22864][core] Disable allocation schedule in Execu...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20050 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20117 @srowen @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira ...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20107 https://github.com/apache/spark/pull/20117 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assi...

2017-12-29 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20117 [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning Small bug fix from last pr, ran a successful merge with this code. You can merge this pull request into a Git repository by running

[GitHub] spark issue #20011: [SPARK-20654][core] Add config to limit disk usage of th...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20011 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira ...

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20107 whoops this is actually broken -- I didn't push exactly what I had locally :( will send a follow up. --- - To unsubscribe, e

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20039#discussion_r159080455 --- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala --- @@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20039#discussion_r159075179 --- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala --- @@ -125,13 +128,39 @@ private[spark] class LiveListenerBus(conf

[GitHub] spark pull request #20039: [SPARK-22850][core] Ensure queued events are deli...

2017-12-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20039#discussion_r159074536 --- Diff: core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala --- @@ -149,7 +158,11 @@ private[spark] class LiveListenerBus(conf: SparkConf

[GitHub] spark issue #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira ...

2017-12-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20107 I've just done some light testing on this so far -- lemme test it some more before we merge it. But thought I'd let folks try it out since it seems to be working for me

[GitHub] spark pull request #20107: [SPARK-22921][PROJECT-INFRA] Choices for Assignin...

2017-12-28 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20107 [SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira on Merge In general jiras are assigned to the original reporter or one of the commentors. This updates the merge script to give you

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19591 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20038: [SPARK-22836][ui] Show driver logs in UI when available.

2017-12-28 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20038 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2017-12-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17619 @andreimaximov that is still sort of the case for all cluster managers. You shouldn't get starvation, you should see the app actively fail (SPARK-15865 was the main change, though some small follow

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19848 +1 to Marcelo's comment about having this conversation somewhere archived. I actually feel like this is something hadoop should be documenting ... we are talking about how committers we

[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2017-12-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17619 ok I think I understand. This sounds like the equivalent of some of the existing blacklisting behavior which current only exists on yarn -- when a request is made to yarn, the spark context tells

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158568388 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerDiskManager.scala --- @@ -0,0 +1,310 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158560723 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158523678 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158517432 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158521861 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158515991 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158510545 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158513454 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158390726 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/DiskStoreManager.scala --- @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158516782 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #20011: [SPARK-20654][core] Add config to limit disk usag...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20011#discussion_r158514239 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/DiskStoreManagerSuite.scala --- @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-22 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158509926 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #20038: [SPARK-22836][ui] Show driver logs in UI when available.

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20038 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 a, got it. sorry I misunderstood the comments on the jira ... I thought your second comment meant the original description was incorrect. anyway -- lgtm

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 that explains how it gets into the list of executors in the UI -- but where does it get the link for the logs

[GitHub] spark issue #20045: [Spark-22360][SQL] Add unit tests for Window Specificati...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20045 failure seems to be unrelated ... Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 this change looks fine -- but I'm having trouble connecting this to the original bug. I guess really I'm confused about how driver logs ever worked ... where is the SparkListenerExecutorAdded event

[GitHub] spark issue #19591: [SPARK-11035][core] Add in-process Spark app launcher.

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19591 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20046 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20045: [Spark-22360][SQL] Add unit tests for Window Specificati...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20045 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/17619 Sorry I am only just looking at this now -- I am not so sure this is doing what you think. I think the notion of "task" in MesosCoarseGrainedSchedulerBackend might be something

[GitHub] spark issue #20047: [SPARK-22861][SQL] SQLAppStatusListener handles multi-jo...

2017-12-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20047 @vanzin @carsonwang --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #20047: [SPARK-22861][SQL] SQLAppStatusListener handles m...

2017-12-21 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20047 [SPARK-22861][SQL] SQLAppStatusListener handles multi-job executions. When one execution has multiple jobs, we need to append to the set of stages, not replace them on every job. Added

[GitHub] spark pull request #19681: [SPARK-20652][sql] Store SQL UI data in the new a...

2017-12-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19681#discussion_r158326122 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala --- @@ -0,0 +1,366 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158316971 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -210,8 +209,13 @@ int getPort() { * Removes the client handle

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-21 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158307838 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -210,8 +209,13 @@ int getPort() { * Removes the client handle

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158198274 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158201917 --- Diff: launcher/src/main/java/org/apache/spark/launcher/package-info.java --- @@ -49,6 +49,15 @@ * * * + * Applications can also

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158201527 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -260,24 +264,30 @@ private long getConnectionTimeout

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158200240 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158200059 --- Diff: launcher/src/main/java/org/apache/spark/launcher/ChildProcAppHandle.java --- @@ -171,10 +90,11 @@ void monitorChild

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158200163 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19591: [SPARK-11035][core] Add in-process Spark app laun...

2017-12-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19591#discussion_r158201743 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -210,8 +209,13 @@ int getPort() { * Removes the client handle

[GitHub] spark issue #19983: [SPARK-22788][streaming] Use correct hadoop config for f...

2017-12-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19983 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19983: [SPARK-22788][streaming] Use correct hadoop config for f...

2017-12-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19983 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19250: [SPARK-12297] Table timezone correction for Times...

2017-12-20 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/19250 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 lgtm @jiangxb1987 are you still looking at this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 ok I just took a look at BroadcastExchangeExec, I see what you mean. It isn't *that* bad, since spark isn't continually creating more instances of those threads (you're not supposed to have more

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 > The last suspicious big group of threads (at least for me) is broadcast-exchange.* but as I've seen this is not false positive because the threadpool never stopped. In BroadcastExchangeExec:

[GitHub] spark pull request #19978: [SPARK-22784][CORE] Configure reading buffer size...

2017-12-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19978#discussion_r157061988 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala --- @@ -72,8 +75,15 @@ private[spark] class ReplayListenerBus extends

[GitHub] spark issue #19978: [SPARK-22784][CORE] Configure reading buffer size in Spa...

2017-12-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19978 agree with @srowen -- how close are we to your optimal size if we just hardcode it to 1MB? I don't think we want the default to be 30MB. btw, what is in your event logs that one line is 1.5

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r157046378 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -318,24 +319,23 @@ private[history] class FsHistoryProvider

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19848 i dunno what the requirements are -- I was hoping you would know the hadoop committer semantics better than me! I suppose a uuid is really the only get something globally unique, as you could even

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r156803714 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r156803613 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark issue #19956: [SPARK-22764][core] Fix flakiness in SparkContextSuite.

2017-12-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19956 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19956: [SPARK-22764][core] Fix flakiness in SparkContextSuite.

2017-12-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19956 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2017-12-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19848 I have one concern about this -- there is a case where you are not giving a unique id to the hadoop committers. You could save one rdd twice, and even have both of those operations running

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156775928 --- Diff: core/src/main/scala/org/apache/spark/status/ElementTrackingStore.scala --- @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156718870 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -772,4 +813,118 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156732445 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -318,24 +319,23 @@ private[history] class FsHistoryProvider

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156706018 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -40,8 +41,10 @@ private[spark] abstract class LiveEntity { var

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156730008 --- Diff: core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala --- @@ -851,6 +842,97 @@ class AppStatusListenerSuite extends

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156716551 --- Diff: core/src/main/scala/org/apache/spark/status/KVUtils.scala --- @@ -68,6 +69,19 @@ private[spark] object KVUtils extends Logging { db

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156710644 --- Diff: core/src/main/scala/org/apache/spark/status/ElementTrackingStore.scala --- @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156709663 --- Diff: core/src/main/scala/org/apache/spark/status/ElementTrackingStore.scala --- @@ -0,0 +1,168 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19751: [SPARK-20653][core] Add cleaning of old elements ...

2017-12-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19751#discussion_r156711461 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -772,4 +813,118 @@ private[spark] class AppStatusListener

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2017-12-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19893 hi I'm just getting back, thanks for looking at this -- one quick comment on the discussion about why there needs to be a whitelist at all. I have a vague memory of looking at this a while back

[GitHub] spark issue #19388: [SPARK-22162] Executors and the driver should use consis...

2017-11-20 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19388 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r152058935 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -298,7 +304,10 @@ private void

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r152055985 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +95,109 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r152053946 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -355,17 +362,33 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r152049670 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -298,7 +304,10 @@ private void

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19769 cc @henryr @zivanfi --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19769: [SPARK-12297][SQL] Adjust timezone for int96 data from i...

2017-11-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19769 thanks for the reviews @cloud-fan @HyukjinKwon @ueshin . I've updated the PR to address all the comments, I believe

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151610021 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -93,13 +94,18 @@ private

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151609609 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala --- @@ -87,4 +96,113 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151609532 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -355,9 +361,31 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151557718 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -298,7 +304,10 @@ private void

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19769#discussion_r151556907 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala --- @@ -355,9 +361,31 @@ class

[GitHub] spark pull request #19769: [SPARK-12297][SQL] Adjust timezone for int96 data...

2017-11-16 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/19769 [SPARK-12297][SQL] Adjust timezone for int96 data from impala ## What changes were proposed in this pull request? Int96 data written by impala vs data written by hive & spark is st

[GitHub] spark issue #19748: [SPARK-20649][core] Simplify REST API resource structure...

2017-11-15 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19748 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19681: [SPARK-20652][sql] Store SQL UI data in the new app stat...

2017-11-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19681 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19748: [SPARK-20649][core] Simplify REST API resource structure...

2017-11-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19748 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19681: [SPARK-20652][sql] Store SQL UI data in the new app stat...

2017-11-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19681 @vanzin merge conflict since I merged the Job & stage page change --- - To unsubscribe, e-mail: reviews-unsu

[GitHub] spark issue #19698: [SPARK-20648][core] Port JobsTab and StageTab to the new...

2017-11-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19698 I merged this to master, since its holding up a bunch more of the history server stuff, but I'd encourage other reviews to take a look still @cloud-fan @ajbozarth @jerryshao

[GitHub] spark issue #19698: [SPARK-20648][core] Port JobsTab and StageTab to the new...

2017-11-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19698 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #19698: [SPARK-20648][core] Port JobsTab and StageTab to ...

2017-11-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19698#discussion_r150655624 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -66,13 +66,8 @@ private[spark] class AppStatusStore(store: KVStore

[GitHub] spark pull request #19698: [SPARK-20648][core] Port JobsTab and StageTab to ...

2017-11-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19698#discussion_r150654983 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -1036,8 +1006,13 @@ private[ui] class TaskDataSource( shuffleRead

[GitHub] spark pull request #19698: [SPARK-20648][core] Port JobsTab and StageTab to ...

2017-11-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19698#discussion_r150637980 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala --- @@ -19,35 +19,52 @@ package org.apache.spark.ui.jobs import

<    5   6   7   8   9   10   11   12   13   14   >