[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode
[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198674#comment-16198674 ] Saisai Shao commented on SPARK-21737: - I was trying to understand how Spark communicate with Mesos, but my knowledge of Mesos is quite poor, till now I cannot figure out a way to address this problem in Mesos. If we only restrict this problem to a Spark on YARN problem, then this is not an issue any more. But my thinking is that only focusing on YARN makes this channel not so useful. Another way is that we build an interface to get endpoint address from cluster manager, and only have a on-yarn implementation initially. Later on we can support Standalone and Mesos if it is required, what do you think? > Create communication channel between arbitrary clients and the Spark AM in > YARN mode > > > Key: SPARK-21737 > URL: https://issues.apache.org/jira/browse/SPARK-21737 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: Jong Yoon Lee >Priority: Minor > > In this JIRA, I develop code to create a communication channel between > arbitrary clients and a Spark AM on YARN. This code can be utilized to send > commands such as getting status command, getting history info from the CLI, > killing the application and pushing new tokens. > Design Doc: > https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode
[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198665#comment-16198665 ] Thomas Graves commented on SPARK-21737: --- I haven't had time to think about it much since the pr discussion. My initial thought was its a generic class/api but you still need resource manager specific parts. I was thinking it would initially only be supported on yarn until someone interested in supporting on the others looked at it. Meaning for yarn you would still use the RM to register the endpoint. Without that you need some other service discovery mechanism like you mention. I'm not a fan of saving this information to hdfs. Have you looked to see if mesos has a similar service? > Create communication channel between arbitrary clients and the Spark AM in > YARN mode > > > Key: SPARK-21737 > URL: https://issues.apache.org/jira/browse/SPARK-21737 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: Jong Yoon Lee >Priority: Minor > > In this JIRA, I develop code to create a communication channel between > arbitrary clients and a Spark AM on YARN. This code can be utilized to send > commands such as getting status command, getting history info from the CLI, > killing the application and pushing new tokens. > Design Doc: > https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode
[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198204#comment-16198204 ] Saisai Shao commented on SPARK-21737: - Hi [~tgraves], I'm trying to understand the design of this. As discussed in PR we planned to change to create a generic client to driver communication channel instead of client-to-AM one. But this raising a question as how to find out RPC endpoint? In this PR, because it is only targeted to YARN, so it leverages YARN api to report AM RPC host/port to RM, and client could get AM RPC endpoint address from RM requesting. But if we're going to build a generic client-to-driver channel, then how to figure out driver RPC endpoint address? If we only restrict our topic to Spark on YARN area, then same solution can be used to figure out driver RPC address. But how to address this issue when running in Standalone/Mesos mode, seems there's no related solution to figure out it. I thought of different ways like using hdfs save driver RPC address to a file and letting client to read it, but none of them are good enough. So I'd like to hear your suggestion on it, do you have any better solution? Thanks in advance! > Create communication channel between arbitrary clients and the Spark AM in > YARN mode > > > Key: SPARK-21737 > URL: https://issues.apache.org/jira/browse/SPARK-21737 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: Jong Yoon Lee >Priority: Minor > > In this JIRA, I develop code to create a communication channel between > arbitrary clients and a Spark AM on YARN. This code can be utilized to send > commands such as getting status command, getting history info from the CLI, > killing the application and pushing new tokens. > Design Doc: > https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode
[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184120#comment-16184120 ] Thomas Graves commented on SPARK-21737: --- We definitely still want to do this. I don't think [~yoonlee95] has time currently, I'm hoping to pick it back up. If someone else has time feel free. > Create communication channel between arbitrary clients and the Spark AM in > YARN mode > > > Key: SPARK-21737 > URL: https://issues.apache.org/jira/browse/SPARK-21737 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: Jong Yoon Lee >Priority: Minor > > In this JIRA, I develop code to create a communication channel between > arbitrary clients and a Spark AM on YARN. This code can be utilized to send > commands such as getting status command, getting history info from the CLI, > killing the application and pushing new tokens. > Design Doc: > https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode
[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183563#comment-16183563 ] Saisai Shao commented on SPARK-21737: - [~yoonlee95], are you still working on this thing? I think this JIRA should still be valid. > Create communication channel between arbitrary clients and the Spark AM in > YARN mode > > > Key: SPARK-21737 > URL: https://issues.apache.org/jira/browse/SPARK-21737 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: Jong Yoon Lee >Priority: Minor > > In this JIRA, I develop code to create a communication channel between > arbitrary clients and a Spark AM on YARN. This code can be utilized to send > commands such as getting status command, getting history info from the CLI, > killing the application and pushing new tokens. > Design Doc: > https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode
[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144034#comment-16144034 ] Apache Spark commented on SPARK-21737: -- User 'yoonlee95' has created a pull request for this issue: https://github.com/apache/spark/pull/18978 > Create communication channel between arbitrary clients and the Spark AM in > YARN mode > > > Key: SPARK-21737 > URL: https://issues.apache.org/jira/browse/SPARK-21737 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.1 >Reporter: Jong Yoon Lee >Priority: Minor > > In this JIRA, I develop code to create a communication channel between > arbitrary clients and a Spark AM on YARN. This code can be utilized to send > commands such as getting status command, getting history info from the CLI, > killing the application and pushing new tokens. > Design Doc: > https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
