[GitHub] spark pull request: [SPARK-9708] [MESOS] Spark should create local...

2015-09-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8358#discussion_r39530770 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -655,15 +655,19 @@ private[spark] object Utils extends Logging { // created

[GitHub] spark pull request: [SPARK-8167] Make tasks that fail from YARN pr...

2015-09-10 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8007#discussion_r39194384 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -390,7 +390,7 @@ private[spark] class

[GitHub] spark pull request: [SPARK-10471] [CORE] [MESOS] prevent getting o...

2015-09-10 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8639#discussion_r39193893 --- Diff: docs/running-on-mesos.md --- @@ -194,6 +194,10 @@ conf.set("spark.mesos.constraints", "tachyon=true;us-east-1=false")

[GitHub] spark pull request: [SPARK-6350][Mesos] Fine-grained mode schedule...

2015-09-09 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8653#discussion_r39019047 --- Diff: core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendSuite.scala --- @@ -42,6 +42,38 @@ import org.apache.spark

[GitHub] spark pull request: [SPARK-6350][Mesos] Fine-grained mode schedule...

2015-09-09 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8653#discussion_r39018501 --- Diff: core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendSuite.scala --- @@ -42,6 +42,38 @@ import org.apache.spark

[GitHub] spark pull request: [SPARK-9708] [MESOS] Spark should create local...

2015-09-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8358#issuecomment-138825979 @dragos what previous parameters are you referring to? SPARK_EXECUTOR_DIRS? I think this logic actually makes sense, as we (Mesos) set the MESOS_DIRECTORY whenever

[GitHub] spark pull request: [SPARK-10471] [CORE] [MESOS] prevent getting o...

2015-09-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8639#issuecomment-138821818 I think the change makes sense, we're planning to add dynamic attribute changes on the slave but that's not merged yet in Mesos. as @dragos mentioned please a

[GitHub] spark pull request: [SPARK-10471] [CORE] [MESOS] prevent getting o...

2015-09-09 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8639#discussion_r39017029 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -240,48 +241,56 @@ private[spark] class

[GitHub] spark pull request: [SPARK-10471] [CORE] [MESOS] prevent getting o...

2015-09-09 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8639#discussion_r39016904 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -56,6 +56,7 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6350][Mesos] Fine-grained mode schedule...

2015-09-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8653#issuecomment-138819813 Good catch! Indeed a regression... this PR LGTM @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-6350][Mesos] Fine-grained mode schedule...

2015-09-09 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/8653#discussion_r39016695 --- Diff: core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendSuite.scala --- @@ -42,6 +42,38 @@ import org.apache.spark

[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...

2015-09-04 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-137845512 Thanks @andrewor14! I've also updated the coarse grain multiple executor patch now as well. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-09-01 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4027#discussion_r38496861 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -357,30 +380,34 @@ private[spark] class

[GitHub] spark pull request: [SPARK-9672][MESOS] Don’t include SPARK_ENV_...

2015-09-01 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7979#issuecomment-136876411 This LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...

2015-08-22 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-133715801 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...

2015-08-21 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-133533197 @andrewor14 PTAL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...

2015-08-20 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/8349 [SPARK-9669][MESOS] Support PySpark on Mesos cluster mode. Support running pyspark with cluster mode on Mesos! This doesn't upload any scripts, so if running in a remote Mesos requires the

[GitHub] spark pull request: [SPARK-10124] [MESOS] Fix removing queued driv...

2015-08-19 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8322#issuecomment-132848583 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-10124] [MESOS] Fix removing queued driv...

2015-08-19 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8322#issuecomment-132815064 @andrewor14 PTAL! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-10124] [MESOS] Fix removing queued driv...

2015-08-19 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/8322 [SPARK-10124] [MESOS] Fix removing queued driver in mesos cluster mode. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tnachen/spark

[GitHub] spark pull request: [SPARK-9672][MESOS] Don’t include SPARK_ENV_...

2015-08-11 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7979#issuecomment-129951341 Can we not filter anything MESOS_ as well? Users set the MESOS_NATIVE_JAVA_LIBRARY in spark_env.sh and this could filter this out --- If your project is set up for it

[GitHub] spark pull request: [SPARK-9645] [yarn] [core] Allow shuffle servi...

2015-08-06 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7966#issuecomment-128512205 Ah got it I thought the chmod is needed but just removed for YARN since YARN manages the directory. Yes I'll create a new ticket for Mesos seperately, then I

[GitHub] spark pull request: [SPARK-9645] [yarn] [core] Allow shuffle servi...

2015-08-06 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7966#issuecomment-128507132 Yes Mesos creates a directory for each task it runs. And if you run any application with Mesos your current directory should be in that directory, or else it's acc

[GitHub] spark pull request: [SPARK-9645] [yarn] [core] Allow shuffle servi...

2015-08-06 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7966#issuecomment-128499976 @andrewor14 I just took a look and I see that it's creating a local directory that's not part of Mesos sandbox. I think we should definitely fix it so it crea

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-08-05 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7820#issuecomment-128186009 This is already merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-08-05 Thread tnachen
Github user tnachen closed the pull request at: https://github.com/apache/spark/pull/7820 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-9645] [yarn] [core] Allow shuffle servi...

2015-08-05 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7966#issuecomment-128183896 Mesos also chown the sandbox directory for you, but I'm not sure where this Utils.getConfiguredLocalDirs directories are going to be created. IIUC Spark

[GitHub] spark pull request: [SPARK-9575][MESOS] Add docuemntation around M...

2015-08-03 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7907#issuecomment-127446325 @andrewor14 updated the doc now, PTAL --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [MESOS] Add docuemntation around Mesos shuffle...

2015-08-03 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7907#issuecomment-127417828 You mean this feature as a whole? :) or just how we incorporated it into Spark on Mesos? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [MESOS] Add docuemntation around Mesos shuffle...

2015-08-03 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/7907 [MESOS] Add docuemntation around Mesos shuffle service. @andrewor14 You can merge this pull request into a Git repository by running: $ git pull https://github.com/tnachen/spark mesos_shuffle

[GitHub] spark pull request: [SPARK-8873][MESOS] Clean up shuffle files if ...

2015-08-02 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7881#issuecomment-127141843 @andrewor14 just tried out your branch and it works on a 1 master 1 slave mesos cluster. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-08-02 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7820#issuecomment-127083197 Yes I tested with a single master and slave and it worked. I can get to the comments after 7 today. > On Aug 2, 2015, at 4:12 PM, andrewor14 wr

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-08-02 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/7820#discussion_r36040263 --- Diff: sbin/start-mesos-shuffle-service.sh --- @@ -0,0 +1,33 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-08-01 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/7820#discussion_r36040223 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -261,7 +269,27 @@ private[spark] class

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-08-01 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/7820#discussion_r36037301 --- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala --- @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-08-01 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/7820#discussion_r36035192 --- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/MesosExternalShuffleService.scala --- @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [MESOS][SPARK-8873] Clean up external shuffle ...

2015-07-31 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/7820 [MESOS][SPARK-8873] Clean up external shuffle data on driver exit with Mesos. @andrewor14 @dragos You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] spark pull request: [SPARK-5095][MESOS] Support capping cores and ...

2015-07-17 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4027#issuecomment-122458457 @andrewor14 that's true, I'm thinking I should work on the removing the shuffle data first since that can help make sure we tie up the dynamic allocation fe

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-15 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-121791952 @andrewor14 I think I figured out what's going on with the test. It's hard to figure out from the logs since it was just hitting a System.exit(1). Hopefully

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-14 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-121167263 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-10 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-120583285 Oh odd I didn't see that in the test output, just saw thrift test returned -1. Let me try to repro. > On Jul 10, 2015, at 10:25 PM, andrewor1

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-10 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-120564418 @andrewor14 I can't seem to get the tests to pass, even though I don't think it's related? --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-10 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-120553564 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-10 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-120281736 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-120237364 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-120191950 @andrewor14 Just rebased --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-120174410 @andrewor14 Just tested the fw auth and roles, and it works on 3 nodes mesos cluster --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-07-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4984#issuecomment-119881649 SPARK-8873 created for cleaning up shuffle files --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-07-06 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4960#discussion_r33996435 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -38,14 +41,51 @@ private[mesos] trait

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-07-06 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5563#issuecomment-118956330 LGTM as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-07-03 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5563#issuecomment-118426045 One more thing after looking at the mesos code more closely (haven't really looked and touched attributes at all while working on mesos), we basically support r

[GitHub] spark pull request: [MESOS][SPARK-8798] Allow additional uris to b...

2015-07-02 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7195#issuecomment-118152614 jenkins retest please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [MESOS][SPARK-8798] Allow additional uris to b...

2015-07-02 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/7195#issuecomment-118112636 @andrewor14 :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [MESOS][SPARK-8798] Allow additional uris to b...

2015-07-02 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/7195 [MESOS][SPARK-8798] Allow additional uris to be fetched with mesos Some users like to download additional files in their sandbox that they can refer to from their spark program, or even later mount

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-06-30 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5563#issuecomment-117057983 @ankurcha You are right this is simply a out of date documentation, and is basically expected to support either a numeric value, set, range or than default to text

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-06-29 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5563#issuecomment-116763524 @squito Thanks for reviewing and commenting, sorry I wasn't really clear I also don't think anything about Mesos should be merged without being tested with a

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-06-25 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5563#issuecomment-115430408 Able to write tests aganist a real mesos cluster is a outstanding issue, which we need more help to setup some automated way to do this. I'm chatting with @drag

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-06-24 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r33200289 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -171,12 +173,15 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-06-09 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-110421459 @andrewor14 Can you look at this when oyu have time? It's been stale for a month now --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-06-06 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r31874085 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -66,6 +65,9 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-06-06 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5563#issuecomment-109695687 @andrewor14 everything else LGTM, can you take a look at this as well? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [MESOS][SPARK-8083] Use the correct base path ...

2015-06-03 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/6615 [MESOS][SPARK-8083] Use the correct base path in mesos driver page. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tnachen/spark

[GitHub] spark pull request: [SPARK-7962][MESOS] Fix master url parsing in ...

2015-05-29 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/6517 [SPARK-7962][MESOS] Fix master url parsing in rest submission client. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tnachen/spark

[GitHub] spark pull request: [SPARK-7962][MESOS] Fix master url parsing in ...

2015-05-29 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/6517#issuecomment-106993604 @andrewor14 PTAL, this is against master branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [MESOS] Fix master url prefix stripping in spa...

2015-05-29 Thread tnachen
Github user tnachen closed the pull request at: https://github.com/apache/spark/pull/6512 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [MESOS] Fix master url prefix stripping in spa...

2015-05-29 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/6512#discussion_r31376509 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -277,8 +277,8 @@ private[spark] class RestSubmissionClient

[GitHub] spark pull request: [MESOS] Fix master url prefix stripping in spa...

2015-05-29 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/6512#issuecomment-106966534 @andrewor14 Mesos cluster mode won't work with 1.4 if this is not merged, I created this commit based on the branch-1.4. Let me know how we can get this in. -

[GitHub] spark pull request: Fix master url prefix stripping in spark submi...

2015-05-29 Thread tnachen
GitHub user tnachen opened a pull request: https://github.com/apache/spark/pull/6512 Fix master url prefix stripping in spark submit. The mesos:// prefix no longer works with SparkSubmit cluster mode, due to a recent change that only strips the master url prefix once. You can

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-18 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30531384 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -86,10 +89,115 @@ private[mesos] trait

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-18 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30531345 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -86,10 +89,115 @@ private[mesos] trait

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-18 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30531198 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -59,6 +58,9 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-18 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30531168 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -86,10 +89,115 @@ private[mesos] trait

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-17 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30473088 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -86,10 +90,101 @@ private[mesos] trait

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-16 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30467685 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -86,10 +90,101 @@ private[mesos] trait

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-16 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30467510 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -86,10 +90,101 @@ private[mesos] trait

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-16 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30467506 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala --- @@ -17,16 +17,20 @@ package

[GitHub] spark pull request: [SPARK-6707] [CORE][MESOS]: Mesos Scheduler sh...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391442 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosUtils.scala --- @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391286 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosUtils.scala --- @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391251 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosUtils.scala --- @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391233 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosUtils.scala --- @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391243 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosUtils.scala --- @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391202 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosUtils.scala --- @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391195 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosUtils.scala --- @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391157 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -20,6 +20,7 @@ package

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391165 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -29,9 +30,15 @@ import org.apache.mesos.{Scheduler

[GitHub] spark pull request: [SPARK-6707] [core]: Mesos Scheduler should al...

2015-05-15 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5563#discussion_r30391110 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -20,16 +20,16 @@ package

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-14 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4984#issuecomment-102150629 Here is the spark settings: spark.executor.uri=http://10.165.143.201:8000/spark-1.4.0-SNAPSHOT-bin-2.2.0.tgz spark.mesos.coarse=true

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-14 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4984#issuecomment-102129458 I verified that the assertion is gone but I'm still getting the old error that I saw before: Caused by: java.lang.RuntimeException: java.lang.RuntimeExce

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-11 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4984#issuecomment-100978624 Oh nothing should have changed too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-11 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4984#issuecomment-100978124 Interesting, we shouldn't actually try to load the library since it might not be present in their machine for everyone. I wonder how come no one reported this b

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-11 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4984#discussion_r30055676 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -130,7 +142,7 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-11 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4984#discussion_r30053918 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -281,4 +319,38 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-11 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4984#discussion_r30053695 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -130,7 +142,7 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-10 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4984#discussion_r30004702 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -281,4 +319,40 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-10 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4984#discussion_r30004596 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -130,7 +142,7 @@ private[spark] class

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-05-07 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-100100864 Not sure what's the right way to do this, but it failed because of a non-related problem. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-6284][MESOS] Add mesos role, principal ...

2015-05-07 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/4960#issuecomment-100100800 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-6287][MESOS] Add dynamic allocation to ...

2015-05-07 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/4984#discussion_r29894844 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala --- @@ -19,6 +19,9 @@ package

[GitHub] spark pull request: [SPARK-7373][MESOS] Add docker support for lau...

2015-05-06 Thread tnachen
Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/5917#discussion_r29804803 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -370,16 +370,30 @@ private[spark] class

[GitHub] spark pull request: [SPARK-7373][MESOS] Add docker support for lau...

2015-05-06 Thread tnachen
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/5917#issuecomment-99519720 @andrewor14 yes this is what is running in our cluster right now, PTAL --- If your project is set up for it, you can reply to this email and have your reply appear on

<    1   2   3   4   5   6   7   >