[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165764166 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2399,6 +2424,115 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165763669 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2399,6 +2424,115 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165761754 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2399,6 +2424,115 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165761207 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2399,6 +2424,115 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165763018 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2399,6 +2424,115 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165764342 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2399,6 +2424,115 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165763274 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -2399,6 +2424,115 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] spark pull request #20244: [SPARK-23053][CORE] taskBinarySerialization and t...

2018-02-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20244#discussion_r165759800 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1016,15 +1016,23 @@ class DAGScheduler( // might modify state

[GitHub] spark issue #20462: [SPARK-23020][core] Fix another race in the in-process l...

2018-02-02 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20462 bq. Is there a release notes / ki kinda thing for Spark releases? not that I know of -- I was just thinking of putting it in the jira, I think that is the best things users have to search

[GitHub] spark issue #20462: [SPARK-23020][core] Fix another race in the in-process l...

2018-02-02 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20462 lgtm The fix here makes sense to me, I see how it breaks the test. I'm just wondering, do we need to doc this at all for users, eg. just clearly describe it in jira? I realize most users

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-02-02 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20422 merged to master. thanks @yaooqinn for doing and updating the tests too --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data

2018-02-02 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19041 Thought some more about the race between `RemoveBlock` getting sent back from the executor vs when the `CacheRecoveryManager` tries to replicate the next block -- actually why is there the back

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165482549 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -420,63 +432,53 @@ private[spark] class ExecutorAllocationManager

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165488466 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165490041 --- Diff: core/src/test/scala/org/apache/spark/CacheRecoveryManagerSuite.scala --- @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165486679 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint

[GitHub] spark issue #20474: [SPARK-23235][Core] Add executor Threaddump to api

2018-02-01 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20474 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20474: [SPARK-23235][Core] Add executor Threaddump to api

2018-02-01 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20474 Jenkins, add to whitelist --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20474: [SPARK-23235][Core] Add executor Threaddump to ap...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20474#discussion_r165379694 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -51,6 +52,21 @@ private[v1] class

[GitHub] spark pull request #20474: [SPARK-23235][Core] Add executor Threaddump to ap...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20474#discussion_r165377861 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -51,6 +52,21 @@ private[v1] class

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-02-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165256701 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -89,26 +96,39 @@ class

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165259010 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint

[GitHub] spark pull request #20424: [Spark-23240][python] Better error message when e...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20424#discussion_r165255299 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala --- @@ -191,7 +191,20 @@ private[spark] class PythonWorkerFactory

[GitHub] spark issue #20461: [SPARK-23289][CORE]OneForOneBlockFetcher.DownloadCallbac...

2018-01-31 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20461 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r165214410 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java --- @@ -126,4 +150,38 @@ private void

[GitHub] spark pull request #20461: [SPARK-23289][CORE]OneForOneBlockFetcher.Download...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20461#discussion_r165213755 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java --- @@ -171,7 +171,9 @@ private void

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165207476 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165207798 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165210078 --- Diff: core/src/main/scala/org/apache/spark/CacheRecoveryManager.scala --- @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/16989#discussion_r165178844 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java --- @@ -126,4 +150,38 @@ private void

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165161260 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -133,4 +133,65 @@ class

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165159188 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -133,4 +133,65 @@ class

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20422 @jerryshao are you ok with making this change? I think our original comments corssed paths as I was taking a closer look

[GitHub] spark pull request #20424: [Spark-23240][python] Better error message when e...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20424#discussion_r165116866 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala --- @@ -191,7 +191,20 @@ private[spark] class PythonWorkerFactory

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164874798 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -423,7 +425,8 @@ private class LiveStage extends LiveEntity

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164886701 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -228,9 +230,12 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164824439 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -594,12 +606,24 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164774722 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage.js --- @@ -416,8 +422,7 @@ $(document).ready(function

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164766057 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage.js --- @@ -416,8 +422,7 @@ $(document).ready(function

[GitHub] spark pull request #20424: [Spark-23240][python] Better error message when e...

2018-01-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20424#discussion_r164765247 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala --- @@ -191,7 +191,20 @@ private[spark] class PythonWorkerFactory

[GitHub] spark pull request #20424: [Spark-23240][python] Better error message when e...

2018-01-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20424#discussion_r164637553 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala --- @@ -191,7 +191,20 @@ private[spark] class PythonWorkerFactory

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r164635886 --- Diff: core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala --- @@ -166,8 +153,20 @@ private[spark] class

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20422 Jenkins, ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20422 thanks for taking a look at this @yaooqinn . To clarify -- there is no bug you are trying to fix here, is there? Its just an optimization? From a quick glance I think the change seems correct

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164627546 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -254,6 +255,7 @@ private class LiveExecutor(val executorId: String, _addTime

[GitHub] spark issue #20408: [SPARK-23189][Core][Web UI] Reflect stage level blacklis...

2018-01-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20408 code looks good to me, but lets what @tgravescs @ajbozarth say. @ajbozarth it is wordy, but I think `Active (Blacklisted in Stages: [...])` is probably the best of the options so far

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164584678 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -594,12 +606,24 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164588595 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -254,6 +255,7 @@ private class LiveExecutor(val executorId: String, _addTime

[GitHub] spark issue #20161: [SPARK-21525][streaming] Check error code from superviso...

2018-01-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20161 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20399: [SPARK-23209][core] Allow credential manager to work whe...

2018-01-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20399 I am merging this now to master & 2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comm

[GitHub] spark pull request #20399: [SPARK-23209][core] Allow credential manager to w...

2018-01-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20399#discussion_r164223721 --- Diff: core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala --- @@ -75,6 +75,17 @@ private[spark] class

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164177919 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -594,12 +606,24 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #20408: [SPARK-23189][Core][Web UI] Reflect stage level b...

2018-01-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20408#discussion_r164175040 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage.js --- @@ -25,9 +25,13 @@ function getThreadDumpEnabled() { return

[GitHub] spark pull request #20399: [SPARK-23209][core] Allow credential manager to w...

2018-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20399#discussion_r163989326 --- Diff: core/src/test/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManagerSuite.scala --- @@ -110,7 +111,64 @@ class

[GitHub] spark pull request #20399: [SPARK-23209][core] Allow credential manager to w...

2018-01-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20399#discussion_r163980133 --- Diff: core/src/test/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManagerSuite.scala --- @@ -110,7 +111,64 @@ class

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 merged to master. thanks @attilapiros --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 @attilapiros can you please update the PR description to also mention node blacklisting (at least briefly) and file a jira for the followup work, and ping me & tom on it?

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 I tried this out on a cluster and seemed fine. I also tried with a bad app, where all tasks fail, its not just because of the host, and all the executors show up as blacklisted, but I guess

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

2018-01-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19992 thanks for looking into this @superbobry -- can you actually close this yourself? we can't directly close it (there is a way but its more complicated

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 @attilapiros test failures look real (you probably just need to regenerate some of those expectations). --- - To unsubscribe, e

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 btw another way you could test out having a bad host would be something like this (untested): ```scala import org.apache.spark.SparkEnv val hosts = sc.parallelize(1 to 1

[GitHub] spark pull request #20203: [SPARK-22577] [core] executor page blacklist stat...

2018-01-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20203#discussion_r162714257 --- Diff: core/src/test/scala/org/apache/spark/scheduler/TaskSetBlacklistSuite.scala --- @@ -59,31 +60,55 @@ class TaskSetBlacklistSuite extends

[GitHub] spark issue #20284: [SPARK-23103][core] Ensure correct sort order for negati...

2018-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20284 even though we don't *know* of this causing a bug in 2.3, I still think we should merge it in there just because there may be some case we aren't thinking of, and this is a relatively small, safe

[GitHub] spark issue #20138: [SPARK-20664][core] Delete stale application data from S...

2018-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20138 as RC1 failed and RC2 is going to be cut soon, I'm going to merge this to master & 2.3 --- - To unsubscribe, e-mail: rev

[GitHub] spark issue #20319: [SPARK-22884][ML][TESTS] ML test for StructuredStreaming...

2018-01-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20319 Jenkins, add to whitelist --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20284: [SPARK-23103][core] Ensure correct sort order for...

2018-01-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20284#discussion_r162166090 --- Diff: common/kvstore/src/test/java/org/apache/spark/util/kvstore/DBIteratorSuite.java --- @@ -112,7 +114,8 @@ public void setup() throws Exception

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 that sounds fine with me --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 yeah I think its probably fine to update the executors page that way -- lets at least see how it looks. Would the list include all stages ever blacklisted? Only those stages still running

[GitHub] spark pull request #20203: [SPARK-22577] [core] executor page blacklist stat...

2018-01-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20203#discussion_r162087946 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala --- @@ -128,13 +130,17 @@ private[scheduler] class TaskSetBlacklist(val conf

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 attila attached some screenshots to the jira (showing both what happens with stage blacklisting and full application blacklisting). The only change here is to the page for a specific stage

[GitHub] spark issue #20138: [SPARK-20664][core] Delete stale application data from S...

2018-01-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20138 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20203 @ajbozarth maybe you have some thoughts on the UI, and whether it makes sense to put anything on the executors page? @CodingCat you also often have good UI suggestions :) thanks

[GitHub] spark pull request #20203: [SPARK-22577] [core] executor page blacklist stat...

2018-01-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20203#discussion_r161884194 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala --- @@ -36,7 +36,9 @@ import org.apache.spark.util.Clock

[GitHub] spark pull request #20203: [SPARK-22577] [core] executor page blacklist stat...

2018-01-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20203#discussion_r161885726 --- Diff: core/src/test/resources/HistoryServerExpectations/stage_blacklisting_for_stage_expectation.json --- @@ -0,0 +1,639 @@ +{ --- End diff

[GitHub] spark pull request #20203: [SPARK-22577] [core] executor page blacklist stat...

2018-01-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20203#discussion_r161884916 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala --- @@ -128,13 +130,17 @@ private[scheduler] class TaskSetBlacklist(val conf

[GitHub] spark pull request #20203: [SPARK-22577] [core] executor page blacklist stat...

2018-01-16 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20203#discussion_r161885207 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -211,6 +211,11 @@ private[spark] class AppStatusListener

[GitHub] spark issue #20236: [SPARK-23044] Error handling for jira assignment

2018-01-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20236 @vanzin @jerryshao want to take another look? Now it * filters out "apache spark" * lets you enter an arbitrary id * if there's an error, just prompts again

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-15 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161660926 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala --- @@ -663,6 +665,95 @@ class FsHistoryProviderSuite extends

[GitHub] spark issue #20056: [SPARK-22878] [CORE] Count totalDroppedEvents for LiveLi...

2018-01-15 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20056 I see that `LiveListenerBus.droppedEventsCounter` and `lastReportTimestamp` are unused, so it certainly makes sense to clean them up one way or the other -- but that might mean we should delete them

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161099778 --- Diff: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala --- @@ -663,6 +665,95 @@ class FsHistoryProviderSuite extends

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161099310 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -834,6 +906,9 @@ private[history] case class

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161098356 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -544,73 +621,75 @@ private[history] class FsHistoryProvider

[GitHub] spark issue #20138: [SPARK-20664][core] Delete stale application data from S...

2018-01-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20138 well, perhaps I mis-represented this -- you still need to turn the event log cleaning on explicitly with the old option, "spark.history.fs.cleaner.enabled". This just doesn

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161095561 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -405,49 +404,70 @@ private[history] class FsHistoryProvider

[GitHub] spark pull request #20138: [SPARK-20664][core] Delete stale application data...

2018-01-11 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20138#discussion_r161082423 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -405,49 +404,70 @@ private[history] class FsHistoryProvider

[GitHub] spark issue #20236: [SPARK-23044] Error handling for jira assignment

2018-01-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20236 @vanzin what exactly are you looking for? The one thing which would be easy is letting you write in an arbitrary jira id (no name searching or anything), that sound OK? I guess this bug

[GitHub] spark issue #20236: [SPARK-23044] Error handling for jira assignment

2018-01-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20236 @jerryshao this should fix it, but I don't have anything to merge to test this out -- would appreciate if someone could try it before we merge

[GitHub] spark pull request #20236: [SPARK-23044] Error handling for jira assignment

2018-01-11 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/20236 [SPARK-23044] Error handling for jira assignment ## What changes were proposed in this pull request? In case the selected user isn't a contributor yet, or any other unexpected error, just

[GitHub] spark issue #20056: [SPARK-22878] [CORE] Count totalDroppedEvents for LiveLi...

2018-01-10 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20056 I have the same question as @jiangxb1987 , what is the situation where you'd use this metric? the jira doesn't say either. seems like existing metrics mostly cover

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-05 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20013 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r160017859 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -119,118 +121,115 @@ private class LiveTask( import

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159988995 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159992789 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159990658 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore

[GitHub] spark issue #20082: [SPARK-22897][CORE]: Expose stageAttemptId in TaskContex...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20082 sorry this is a little late, but lgtm too. agree with the points above about leaving the old name deprecated and moving to the new name

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19992 change is fine, but from discussion on the jira I'm unclear if this is really worth it -- gain seems pretty small after the other fix in 2.3

[GitHub] spark pull request #19992: [SPARK-22805][CORE] Use StorageLevel aliases in e...

2018-01-04 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19992#discussion_r159812291 --- Diff: core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala --- @@ -2022,12 +1947,7 @@ private[spark] object JsonProtocolSuite extends

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/19848 @steveloughran can you bring this up on dev@? we should move this discussion off of this PR. (sorry haven't had a chance to look yet, but I appreciate you doing

[GitHub] spark issue #20039: [SPARK-22850][core] Ensure queued events are delivered t...

2018-01-04 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20039 merged to master / 2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20117: [SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning

2017-12-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20117 merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

<    4   5   6   7   8   9   10   11   12   13   >