[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2017-03-23 Thread mccheah
Github user mccheah commented on the issue:

https://github.com/apache/spark/pull/16061
  
The diff showing our progress so far has moved to 
https://github.com/apache-spark-on-k8s/spark/pull/200 for anyone following 
along.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2017-03-03 Thread foxish
Github user foxish commented on the issue:

https://github.com/apache/spark/pull/16061
  
+1 I think we should close this PR to avoid confusion. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2017-03-03 Thread mccheah
Github user mccheah commented on the issue:

https://github.com/apache/spark/pull/16061
  
Work on this feature has moved to 
https://github.com/apache-spark-on-k8s/spark. The exact diff we are working 
with is this: https://github.com/apache-spark-on-k8s/spark/pull/1. Feel free to 
provide feedback on the fork's PR.

@erikerlandson can you close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-12-02 Thread rxin
Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16061
  
We should acknowledge people that have done this in the merged commit, but 
I don't think it'd make a lot of sense to do a merge rather than squash. The 
initial Spark SQL commit was a single commit even though it was a much larger 
project https://github.com/apache/spark/pull/146


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-12-02 Thread erikerlandson
Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/16061
  
@iyanuobidele the current head of this branch should be equivalent, but it 
can't hurt to rebase just in case.

@foxish yesterday occurred to me that it would've been sensible set up a GH 
organization for this little consortium, and run the PR from there.  Presumably 
we still could.   The drawback to making any change to it now is losing the 
dialog that's happened here already.

The problem I'm interested in solving even more is ensuring that everybody 
who contributes to this PR gets logged somehow in the upstream commit history 
when this PR eventually gets merged.   I don't think that would happen in the 
case of a typical squash-merge, although I assume it _would_ using a rebase 
merge. @rxin do you have any thoughts on that? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-12-01 Thread foxish
Github user foxish commented on the issue:

https://github.com/apache/spark/pull/16061
  
I would think that it would be easier if we continue to keep PRs and issues 
in foxish/spark since we have the previous PRs and issues over there. It should 
be easy enough to pick those commits up and update this branch once we review 
and merge them. It doesn't matter where it lives really, except that it seems 
like most of it is already on one place. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-12-01 Thread iyanuobidele
Github user iyanuobidele commented on the issue:

https://github.com/apache/spark/pull/16061
  
@erikerlandson are there any new commits to this branch ? Or it's at the 
same level as @foxish's k8s-support. In case I need to rebase


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-12-01 Thread erikerlandson
Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/16061
  
At the risk of redundant announcements, I am going to present this 
spark-on-kube topic at the [Kubernetes 
SIG-Apps](https://github.com/kubernetes/community/tree/master/sig-apps) next 
Monday (Dec 5).  It will be similar to my earlier [OpenShift 
briefing](https://blog.openshift.com/openshift-commons-big-data-sig-2-running-apache-spark-natively-on-kubernetes-with-openshift/)
 but the focus will be on kube specifically, and I'll update it to cover all 
the latest developments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-12-01 Thread erikerlandson
Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/16061
  
Since we have at least four people with interest in working on the code, I 
have a workflow proposal, which is that we all submit candidate updates to 
#16061 in the form of PRs against 
[erikerlandson:spark-18278-k8s-native](https://github.com/erikerlandson/spark/tree/spark-18278-k8s-native)
 (recursive PRs against a PR!)

I did this with Anirudh against his `k8s-support` branch, and it seemed to 
work fine.  It should help keep asynchronous contributions sane, and provides 
the usual durable public forum for discussing the updates as they evolve.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-30 Thread foxish
Github user foxish commented on the issue:

https://github.com/apache/spark/pull/16061
  
Cross posting link to [the 
proposal](https://issues.apache.org/jira/secure/attachment/12840988/SPARK-18278%20-%20Spark%20on%20Kubernetes%20Design%20Proposal.pdf)
 here. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread ash211
Github user ash211 commented on the issue:

https://github.com/apache/spark/pull/16061
  
Another external scheduler backend I'm aware of is Two Sigma's scheduler 
backend for the system they've created called 
[Cook](https://github.com/twosigma/Cook).  See 
[CoarseCookSchedulerBackend.scala](https://github.com/twosigma/spark/blob/2.0.2-cook/core/src/main/scala/org/apache/spark/scheduler/CoarseCookSchedulerBackend.scala)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread erikerlandson
Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/16061
  
@rxin, when you say "move all resource managers" does that mean "move 
scheduler back-ends for mesos, yarn, etc, into some `resource-managers` 
sub-project"  ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread liancheng
Github user liancheng commented on the issue:

https://github.com/apache/spark/pull/16061
  
@erikerlandson For the RAT failure, you may either add Apache license 
header to newly added files or add the file to `dev/.rat-excludes`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread tnachen
Github user tnachen commented on the issue:

https://github.com/apache/spark/pull/16061
  
@rxin Makes sense, @srowen also talked about starting the discussion of 
having a better support for external cluster managers as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread rxin
Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16061
  
One thing - can we submit a separate pr to move all resource managers into 

resource-managers/yarn
resource-managers/mesos

?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread rxin
Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16061
  
This is pretty cool.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16061
  
**[Test build #69335 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69335/consoleFull)**
 for PR 16061 at commit 
[`8584913`](https://github.com/apache/spark/commit/8584913d51c1f68c8f9d3cdf142947cd35cdfcc2).
 * This patch **fails RAT tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16061
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69335/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16061
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark issue #16061: [SPARK-18278] [Scheduler] Support native submission of s...

2016-11-29 Thread SparkQA
Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16061
  
**[Test build #69335 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69335/consoleFull)**
 for PR 16061 at commit 
[`8584913`](https://github.com/apache/spark/commit/8584913d51c1f68c8f9d3cdf142947cd35cdfcc2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org