[GitHub] spark issue #21899: [SPARK-24912][SQL] Don't obscure source of OOM during br...

2018-08-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21899 lgtm, just a small reword in one of the msgs suggested by maxgekk. @hvanhovell @gatorsmile would you like to review this as well

[GitHub] spark issue #21950: [SPARK-24914][SQL][WIP] Add configuration to avoid OOM d...

2018-08-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21950 restest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22101: [SPARK-25114][Core] Fix RecordBinaryComparator when subt...

2018-08-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/22101 the added tests are good. This is pretty nit-picky, but looking at the whole test suite, are there any tests that check for anything other than the first byte (or array length)? Seems the longer

[GitHub] spark issue #21950: [SPARK-24914][SQL][WIP] Add configuration to avoid OOM d...

2018-08-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21950 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21451: [SPARK-24296][CORE] Replicate large blocks as a stream.

2018-08-21 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21451 @tgravescs @vanzin any more comments? I think I've addressed everything --- - To unsubscribe, e-mail: reviews-uns

[GitHub] spark pull request #22195: [CORE] Fix typo in spark.network.crypto.keyFactor...

2018-08-22 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/22195 [CORE] Fix typo in spark.network.crypto.keyFactoryIterations You can merge this pull request into a Git repository by running: $ git pull https://github.com/squito/spark SPARK-25205

[GitHub] spark pull request #21923: [SPARK-24918][Core] Executor Plugin api

2018-08-22 Thread squito
Github user squito closed the pull request at: https://github.com/apache/spark/pull/21923 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21923: [SPARK-24918][Core] Executor Plugin api

2018-08-22 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21923 this is being continued in https://github.com/apache/spark/pull/22192 --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-07-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21440 @mridulm @jerryshao maybe you would be interested in reviewing this as well? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #21131: [SPARK-23433][CORE] Late zombie task completions update ...

2018-07-11 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21131 >>Fetch failure lead to multiple tasksets which are active for a given stage. >How can this happen? the TaskSetManager will mark itself as zombie when it receives a fetch failed.

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r203097734 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,34 @@ private[spark] class ChunkedByteBuffer(var chunks

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-07-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r203100978 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -160,11 +160,29 @@ case class SparkListenerBlockUpdated

[GitHub] spark issue #21656: [SPARK-24677][Core]Avoid NoSuchElementException from Med...

2018-07-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21656 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21639: [SPARK-24653][tests] Avoid cross-job pollution in TestUt...

2018-07-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21639 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21131: [SPARK-23433][CORE] Late zombie task completions update ...

2018-07-17 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21131 The DAGSCheudler is notified about successfully completed tasks, whether or not the `tsm.successful` is already true: https://github.com/apache/spark/blob/master/core/src/main/scala/org

[GitHub] spark pull request #21593: [SPARK-24578][Core] Cap sub-region's size of retu...

2018-07-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21593#discussion_r203157915 --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java --- @@ -137,30 +137,15 @@ protected void deallocate

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r203244832 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,34 @@ private[spark] class ChunkedByteBuffer(var chunks

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r203245221 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,34 @@ private[spark] class ChunkedByteBuffer(var chunks

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-17 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r20324 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBufferFileRegion.scala --- @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r203381863 --- Diff: core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala --- @@ -166,6 +170,34 @@ private[spark] class ChunkedByteBuffer(var chunks

[GitHub] spark pull request #21131: [SPARK-23433][CORE] Late zombie task completions ...

2018-07-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21131#discussion_r203415036 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -764,6 +769,19 @@ private[spark] class TaskSetManager

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-07-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r203455379 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -160,11 +160,29 @@ case class SparkListenerBlockUpdated

[GitHub] spark issue #21698: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-07-18 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21698 jumping in the middle on this discussion -- everybody has raised some great points. 1) My first takeaway from this is just how hard it can be to reason about this, because spark's

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r203472192 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -659,6 +659,11 @@ private[spark] class BlockManager( * Get block

[GitHub] spark issue #21729: [SPARK-24755][Core] Executor loss can cause task to not ...

2018-07-18 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21729 lgtm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-07-18 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r203520320 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -160,11 +160,29 @@ case class SparkListenerBlockUpdated

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-07-18 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21451 @mridulm @jerryshao @felixcheung last one in the 2GB block limit series. just rebased to include the updates to https://github.com/apache/spark/pull/21440. I will also run my tests on a cluster

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-07-18 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21451 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21440: [SPARK-24307][CORE] Support reading remote cached...

2018-07-19 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21440#discussion_r203773818 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -659,6 +659,11 @@ private[spark] class BlockManager( * Get block

[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...

2018-07-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 Ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...

2018-07-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 can you add something in the PR description about how this is important because sometimes many of these messages queue up in netty's ChannelOutboundBuffer before transferTo() is called?

[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...

2018-07-19 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 @zsxwing @jerryshao @Victsm you might be interested in this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r204171230 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceTypeHelper.scala --- @@ -0,0 +1,119 @@ +/* + * Licensed to the

[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...

2018-07-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 > I like it, but this will still create the byte channel right? is there a way to reuse it? we could create a pool, though management becomes a bit more complex. would you ever shrink

[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...

2018-07-23 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 > Does it make sense to release byteChannel at deallocate()? you could, just to let GC kick in a *bit* earlier, but I don't think its going to make a big difference. (Netty's

[GitHub] spark issue #21440: [SPARK-24307][CORE] Support reading remote cached partit...

2018-07-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21440 @gatorsmile sure, thats pretty easy. I'll submit a follow up pr. --- - To unsubscribe, e-mail: reviews-uns

[GitHub] spark issue #21474: [SPARK-24297][CORE] Fetch-to-disk by default for > 2gb

2018-07-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21474 good point @jerryshao , I've updated the docs now, please take a look, thanks --- - To unsubscribe, e-mail: reviews-uns

[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...

2018-07-24 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 lgtm will wait a bit for any more comments before merging --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #21867: [SPARK-24307][CORE] Add conf to revert to old cod...

2018-07-24 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/21867 [SPARK-24307][CORE] Add conf to revert to old code. In case there are any issues in converting FileSegmentManagedBuffer to ChunkedByteBuffer, add a conf to go back to old code path

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r204912925 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -359,20 +368,56 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r204917880 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -359,20 +366,55 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r204914384 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDDBarrier.scala --- @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-24 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r204917245 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1647,6 +1647,14 @@ abstract class RDD[T: ClassTag

[GitHub] spark pull request #21867: [SPARK-24307][CORE] Add conf to revert to old cod...

2018-07-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21867#discussion_r205124896 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -731,7 +731,14 @@ private[spark] class BlockManager

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r205126592 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -359,20 +366,55 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #21867: [SPARK-24307][CORE] Add conf to revert to old cod...

2018-07-25 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21867#discussion_r205150784 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala --- @@ -731,7 +733,14 @@ private[spark] class BlockManager

[GitHub] spark issue #21698: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-07-25 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21698 sorry I got bogged down in some other things, thanks for the responses: >> on a fetch-failure in repartition, fail the entire job > Currently I can't figure out

[GitHub] spark issue #21451: [SPARK-24296][CORE][WIP] Replicate large blocks as a str...

2018-07-25 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21451 fyi, I did finally run my scale tests again on a cluster, and shuffles, remote reads, and replication worked for blocks over 2gb (sorry got sidetracked with a few other things in the meantime

[GitHub] spark pull request #21880: [SPARK-24929][INFRA] Make merge script don't swal...

2018-07-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21880#discussion_r205470335 --- Diff: dev/merge_spark_pr.py --- @@ -331,6 +331,9 @@ def choose_jira_assignee(issue, asf_jira): assignee = asf_jira.user

[GitHub] spark issue #21474: [SPARK-24297][CORE] Fetch-to-disk by default for > 2gb

2018-07-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21474 oh no, good point Tom. it'll fail with ``` 18/07/26 07:15:02 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 1.3 (TID 15, irashid-large-2.gce.cloudera.com, execu

[GitHub] spark issue #21474: [SPARK-24297][CORE] Fetch-to-disk by default for > 2gb

2018-07-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21474 https://issues.apache.org/jira/browse/SPARK-24936 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #21653: [SPARK-13343] speculative tasks that didn't commit shoul...

2018-07-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21653 I kicked off the test manually at https://spark-prs.appspot.com/users/hthuynh2. I dunno why the test triggering via comments stops workign on some prs

[GitHub] spark issue #21885: [SPARK-24926] Ensure numCores is used consistently in al...

2018-07-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21885 OK to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r205652317 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -359,20 +366,55 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-26 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r205652334 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1647,6 +1647,14 @@ abstract class RDD[T: ClassTag

[GitHub] spark issue #21811: [SPARK-24801][CORE] Avoid memory waste by empty byte[] a...

2018-07-26 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21811 merged to master, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-27 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r205876806 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala --- @@ -359,20 +366,55 @@ private[spark] class TaskSchedulerImpl

[GitHub] spark issue #21899: [SPARK-24912][SQL] Don't obscure source of OOM during br...

2018-07-27 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21899 this looks good -- wondering about a couple of potential improvements. Is it possible to include the actual size of the in-memory table so far in the msg as well? Also, does catching the

[GitHub] spark issue #21908: [MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSui...

2018-07-30 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21908 late lgtm from me too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #21923: [SPARK-24918][Core] Executor Plugin api

2018-07-30 Thread squito
GitHub user squito opened a pull request: https://github.com/apache/spark/pull/21923 [SPARK-24918][Core] Executor Plugin api This provides a very simple api for users to specify arbitrary code to run within an executor, eg. for debugging or added instrumentation. The intial

[GitHub] spark pull request #21899: [SPARK-24912][SQL] Don't obscure source of OOM du...

2018-07-30 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21899#discussion_r206327279 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala --- @@ -118,12 +119,19 @@ case class

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-07-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r206682882 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1349,6 +1339,29 @@ class DAGScheduler( s"l

[GitHub] spark issue #21915: [SPARK-24954][Core] Fail fast on job submit if run a bar...

2018-07-31 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21915 thanks @jiangxb1987, lgtm aside from @mengxr 's comments --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.or

[GitHub] spark issue #21927: [SPARK-24820][Core] Fail fast when submitted job contain...

2018-07-31 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21927 > Second thought: PartitionPruningRDD is just an implementation of RDD. Every user / developer can implement a similar one. Also this doesn't handle the case mentioned by @felixcheung : a

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206695469 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -156,6 +160,14 @@ private[yarn] class YarnAllocator

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206696830 --- Diff: resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala --- @@ -35,18 +36,22 @@ import

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

2018-07-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20761#discussion_r206696107 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -275,9 +287,13 @@ private[yarn] class YarnAllocator

[GitHub] spark issue #21923: [SPARK-24918][Core] Executor Plugin api

2018-08-01 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21923 > Are there more specific use cases? I always feel it'd be impossible to design APIs without seeing couple different use cases. With this basic api, you could just do things that

[GitHub] spark pull request #21927: [SPARK-24820][SPARK-24821][Core] Fail fast when s...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21927#discussion_r206985923 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -340,6 +340,22 @@ class DAGScheduler

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r206995945 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorMetrics.scala --- @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r206998585 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -669,6 +686,34 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r20799 --- Diff: core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala --- @@ -251,6 +261,214 @@ class EventLoggingListenerSuite extends

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207006413 --- Diff: core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207004345 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +103,50 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207004013 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -669,6 +686,34 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207004094 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +103,50 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21923: [SPARK-24918][Core] Executor Plugin api

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21923#discussion_r207036143 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -130,6 +130,12 @@ private[spark] class Executor( private val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r207037118 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -669,6 +686,34 @@ private[spark] class AppStatusListener

[GitHub] spark pull request #21758: [SPARK-24795][CORE] Implement barrier execution m...

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21758#discussion_r207076922 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1349,6 +1339,29 @@ class DAGScheduler( s"l

[GitHub] spark pull request #21923: [SPARK-24918][Core] Executor Plugin api

2018-08-01 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21923#discussion_r207077147 --- Diff: core/src/main/java/org/apache/spark/AbstractExecutorPlugin.java --- @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #21475: [SPARK-24416] Fix configuration specification for killBl...

2018-06-12 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21475 merged to master. Thanks @redsanket ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-12 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21527 I'm totally onboard with exposing it, though I'd say undocumented unless we have some concrete advice on how to set it. I was just explaining my experience so far and why I haven'

[GitHub] spark issue #21558: [SPARK-24552][SQL] Use task ID instead of attempt number...

2018-06-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21558 IMO your change is the right fix, not just a workaround. I don't think its a scheduler bug (though its definitely unclear). I'll move that discussion to the jira. An alternativ

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195291809 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -304,6 +305,11 @@ class SparkContext(config: SparkConf) extends Logging

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195291213 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +101,53 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195290278 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -169,6 +182,31 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195290854 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +101,53 @@ class ExecutorSummary private[spark]( val

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195289142 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -93,6 +96,9 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195289072 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1751,7 +1753,7 @@ class DAGScheduler

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195289751 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -169,6 +182,31 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195290564 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -234,8 +272,18 @@ private[spark] class EventLoggingListener

[GitHub] spark pull request #21221: [SPARK-23429][CORE] Add executor memory metrics t...

2018-06-13 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21221#discussion_r195297024 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala --- @@ -98,14 +101,53 @@ class ExecutorSummary private[spark]( val

[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-06-13 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21221 Jenkins, test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #21221: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-06-14 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21221 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #21346: [SPARK-6237][NETWORK] Network-layer changes to al...

2018-06-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21346#discussion_r195464525 --- Diff: common/network-common/src/main/java/org/apache/spark/network/client/TransportClient.java --- @@ -220,30 +196,91 @@ public long sendRpc(ByteBuffer

[GitHub] spark pull request #21346: [SPARK-6237][NETWORK] Network-layer changes to al...

2018-06-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21346#discussion_r195464933 --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/RpcHandler.java --- @@ -38,15 +38,28 @@ * * This method will

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-06-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r195469104 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,189

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-06-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r195467887 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -0,0 +1,189

[GitHub] spark pull request #21068: [SPARK-16630][YARN] Blacklist a node if executors...

2018-06-14 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21068#discussion_r195566345 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorBlacklistTracker.scala --- @@ -151,9 +149,9 @@ private[spark

[GitHub] spark issue #21068: [SPARK-16630][YARN] Blacklist a node if executors won't ...

2018-06-15 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/21068 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

<    6   7   8   9   10   11   12   13   14   15   >