[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151673115 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -349,13 +349,22

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151672855 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -427,10 +444,10 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r151672564 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -175,17 +176,39 @@ trait

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-11-15 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @srowen @vanzin gentle ping... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-11-14 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @srowen may get I get a merge please? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-07 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r149367812 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -451,15 +465,20 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-11-03 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r148786663 --- Diff: dev/deps/spark-deps-hadoop-2.6 --- @@ -138,7 +138,7 @@ lz4-java-1.4.0.jar machinist_2.11-0.6.1.jar macro-compat_2.11-1.1.1.jar mail

[GitHub] spark issue #19374: [SPARK-22145][MESOS] fix supervise with checkpointing on...

2017-11-02 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19374 @srowen gentle ping. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r147370701 --- Diff: dev/deps/spark-deps-hadoop-2.6 --- @@ -138,7 +138,7 @@ lz4-java-1.4.0.jar machinist_2.11-0.6.1.jar macro-compat_2.11-1.1.1.jar mail

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-27 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r147370285 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -427,10 +441,10 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-26 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r147087721 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -175,17 +176,36 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-26 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r147087477 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -451,15 +465,20 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-26 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r147087405 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -349,13 +349,22

[GitHub] spark pull request #17813: [SPARK-20540][CORE] Fix unstable executor request...

2017-10-25 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/17813#discussion_r146831076 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -589,8 +605,18 @@ class

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @susanxhuynh I will re-test and update the PR, the issue was at proto bufs, using reservation info without checking if the offer had that field anyway

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @susanxhuynh this does not work with statically resources and mesos 1.4. There is a compatibility issue as proto has changed between versions. In the following scenario I have a static

[GitHub] spark issue #19555: [SPARK-22133][DOCS] Documentation for Mesos Reject Offer...

2017-10-23 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19555 @susanxhuynh could you pls review this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19510: [SPARK-22292][Mesos] Added spark.mem.max support for Mes...

2017-10-20 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19510 @susanxhuynh pls review. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #19510: [SPARK-22292][Mesos] Added spark.mem.max support for Mes...

2017-10-20 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19510 @windkit https://github.com/apache/spark/blob/e2fea8cd6058a807ff4841b496ea345ff0553044/docs/running-on-mesos.md

[GitHub] spark pull request #19510: [SPARK-22292][Mesos] Added spark.mem.max support ...

2017-10-20 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19510#discussion_r145936265 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -64,6 +64,7

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r145645133 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -380,7 +389,8

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r145643615 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -446,6 +456,9

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-18 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @srowen could I get a review or a merge pls? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-18 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r145397272 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -380,7 +389,8

[GitHub] spark issue #19374: [SPARK-22145][MESOS] fix supervise with checkpointing on...

2017-10-18 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19374 @srowen Could I get a merge or a review pls? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-18 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r145391350 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -135,22 +135,24 @@ private

[GitHub] spark pull request #19510: [SPARK-22292][Mesos] Added spark.mem.max support ...

2017-10-18 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19510#discussion_r145389586 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -64,6 +64,7

[GitHub] spark issue #19510: [SPARK-22292][Mesos] Added spark.mem.max support for Mes...

2017-10-18 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19510 @windkit there is an open issue here: https://issues.apache.org/jira/browse/SPARK-22133 Could you please add documentation of the new property along with the old ones

[GitHub] spark pull request #19510: [SPARK-22292][Mesos] Added spark.mem.max support ...

2017-10-18 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19510#discussion_r145383146 --- Diff: resource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala --- @@ -152,6

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r145237956 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -380,7 +389,8

[GitHub] spark issue #19374: [SPARK-22145][MESOS] fix supervise with checkpointing on...

2017-10-17 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19374 @susanxhuynh @ArtRand ready for another round. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19374: [SPARK-22145][MESOS] fix supervise with checkpointing on...

2017-10-17 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19374 @susanxhuynh time is correct when using latest spark. If you check the old pics in the description the spark used is 2.1, DC/OS image is old. I updated it and re-run things: ![image](https

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r145079551 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -17,10 +17,14

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r145079127 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala --- @@ -159,7 +160,8

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-17 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r145066047 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -276,8 +276,8 @@ private[spark

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r144947372 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -896,8 +913,8 @@ private[spark

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r144947305 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -374,6 +375,15 @@ private

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r144947255 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -373,10 +374,16 @@ class SparkContext(config: SparkConf) extends Logging

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-16 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @ArtRand I updated the PR check my comments pls. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r144835209 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -469,6 +474,12 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r144835101 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -175,14 +175,22 @@ trait

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r144835048 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -28,6 +28,8 @@ import

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations on ...

2017-10-16 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19390#discussion_r144834956 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -380,7 +389,8

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-12 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 @susanxhuynh could you review pls? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #19468: [SPARK-18278] [Scheduler] Spark on Kubernetes - Basic Sc...

2017-10-11 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19468 + @susanxhuynh @ArtRand (Mesosphere). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-11 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 Glad to hear that @tawfiqul-islam, thank you for the feedback. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-11 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r143969946 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -804,45 +814,52 @@ private

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-11 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r143969875 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -374,6 +375,15 @@ private

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-10-11 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19374#discussion_r143969837 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -276,8 +276,8 @@ private[spark

[GitHub] spark issue #19374: [SPARK-22145][MESOS] fix supervise with checkpointing on...

2017-10-11 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19374 @susanxhuynh good catch I dont know about the time issue. Seems weird, maybe another bug? This is the section of the not completed apps by the way since this is a streaming app

[GitHub] spark issue #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-11 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19437 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-09 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19437 @susanxhuynh this a mesos 1.4 feature, shouldn't we document this for users? https://issues.apache.org/jira/browse/MESOS-7418

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-09 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r143415419 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -170,9 +174,122 @@ private

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-09 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r143413006 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala --- @@ -21,6 +21,39 @@ import java.util.concurrent.TimeUnit

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-09 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r143411366 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -170,9 +174,122 @@ private

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-06 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r143312903 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -17,10 +17,14

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-06 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r143312857 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -122,7 +126,7 @@ private

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-06 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r143221450 --- Diff: resource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/Utils.scala --- @@ -105,4 +108,108 @@ object Utils { def

[GitHub] spark pull request #19437: [SPARK-22131][MESOS] Mesos driver secrets

2017-10-06 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19437#discussion_r143221337 --- Diff: resource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/Utils.scala --- @@ -105,4 +108,108 @@ object Utils { def

[GitHub] spark issue #19374: [SPARK-22145][MESOS] fix supervise with checkpointing on...

2017-10-05 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19374 @ArtRand @susanxhuynh gentle ping. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #18098: [SPARK-16944][Mesos] Improve data locality when launchin...

2017-10-03 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18098 @ArtRand +1 for the pluggable interface... in general we are planning to optimize resource allocation and locality in another PR, (fenzo form Netflix is something we would like to test which

[GitHub] spark pull request #19390: [SPARK-18935][MESOS] Fix dynamic reservations

2017-09-29 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/19390 [SPARK-18935][MESOS] Fix dynamic reservations ## What changes were proposed in this pull request? - Solves the issue described in the ticket by preserving reservation and allocation info

[GitHub] spark issue #19374: [SPARK-22145][MESOS] fix supervise with checkpointing on...

2017-09-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19374 @ArtRand @susanxhuynh pls review. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #19374: [SPARK-22145][MESOS] fix supervise with checkpoin...

2017-09-27 Thread skonto
GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/19374 [SPARK-22145][MESOS] fix supervise with checkpointing on mesos ## What changes were proposed in this pull request? - Fixes the issue with frameworkId being recovered by checkpointed data

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-20 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r140088442 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCredentialRenewer.scala --- @@ -0,0 +1,150

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139841458 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -159,6 +159,13 @@ class

[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...

2017-09-19 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19272 When executor is started it asks from CoarseSchedulerBackend the spark config which contains the hadoop credentials https://github.com/apache/spark/blob/7c92351f43ac4b1710e3c80c78f7978dad491ed2/core

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139642261 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala --- @@ -194,6 +198,24

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139640414 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCredentialRenewer.scala --- @@ -0,0 +1,150

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139637343 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -159,6 +159,13 @@ class

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139636800 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -174,6 +178,16 @@ private[spark] class

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139635260 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -123,6 +125,8 @@ private[spark] class

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139634133 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCredentialRenewer.scala --- @@ -0,0 +1,150

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139633693 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCredentialRenewer.scala --- @@ -0,0 +1,150

[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...

2017-09-19 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/19272#discussion_r139625400 --- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala --- @@ -174,6 +178,16 @@ private[spark] class

[GitHub] spark issue #7842: [SPARK-8542][MLlib]PMML export for Decision Trees

2017-08-27 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/7842 @coderxiang what is the plan for this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18784: [SPARK-21559][Mesos] remove mesos fine-grained mode

2017-08-22 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18784 @ArtRand @susanxhuynh pls review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-14 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin I dont forgot I didnt see agreement. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-11 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-11 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132765579 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-10 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin do you think we are ready for a merge? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18910: [SPARK-21694][MESOS] Support Mesos CNI network labels

2017-08-10 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18910 @susanxhuynh have you tested how spark jobs behave with a real cni network, just curious should be no difference right? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request #18910: [SPARK-21694][MESOS] Support Mesos CNI network la...

2017-08-10 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18910#discussion_r132581612 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -162,7 +163,11 @@ private

[GitHub] spark pull request #18910: [SPARK-21694][MESOS] Support Mesos CNI network la...

2017-08-10 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18910#discussion_r132580807 --- Diff: resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala --- @@ -21,6 +21,7 @@ import

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-09 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 This is how I build things: ./build/mvn -Pmesos -Phadoop-2.7 -Dhadoop.version=2.7.0 -DskipTests clean package # -DskipTests clean package export JAVA_HOME=/usr/lib/jvm

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-09 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @BryanCutler sure check here, it works: https://gist.github.com/skonto/dc2070d1529c97ec5de32e99983a834f --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-09 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 > spark-2.3.0-SNAPSHOT-bin-18630/bin$ ./spark-shell --verbose --master spark://ip-10-10-1-79:7077 Using properties file: null Parsed arguments: master spark://ip-10

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-09 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @BryanCutler you just started spark shell and it failed? How can I reproduce it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-09 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @BryanCutler @vanzin to make things testable DriverWrapper needs refactoring from a quick look I took. py files are resolved in client mode, let's fix it in another PR (I could do

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-09 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin I fixed these minor issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

2017-08-08 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/18630 @vanzin fixed the issues. Ready for merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132072398 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132072408 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132059601 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +70,24 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132058305 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132058269 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132058214 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132058239 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala --- @@ -66,4 +70,24 @@ object DriverWrapper { System.exit(-1

[GitHub] spark pull request #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for st...

2017-08-08 Thread skonto
Github user skonto commented on a diff in the pull request: https://github.com/apache/spark/pull/18630#discussion_r132056579 --- Diff: core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation

<    1   2   3   4   5   6   7   8   9   10   >