[
https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198204#comment-16198204
]
Saisai Shao commented on SPARK-21737:
-------------------------------------
Hi [~tgraves], I'm trying to understand the design of this. As discussed in PR
we planned to change to create a generic client to driver communication channel
instead of client-to-AM one. But this raising a question as how to find out RPC
endpoint?
In this PR, because it is only targeted to YARN, so it leverages YARN api to
report AM RPC host/port to RM, and client could get AM RPC endpoint address
from RM requesting. But if we're going to build a generic client-to-driver
channel, then how to figure out driver RPC endpoint address?
If we only restrict our topic to Spark on YARN area, then same solution can be
used to figure out driver RPC address. But how to address this issue when
running in Standalone/Mesos mode, seems there's no related solution to figure
out it. I thought of different ways like using hdfs save driver RPC address to
a file and letting client to read it, but none of them are good enough.
So I'd like to hear your suggestion on it, do you have any better solution?
Thanks in advance!
> Create communication channel between arbitrary clients and the Spark AM in
> YARN mode
> ------------------------------------------------------------------------------------
>
> Key: SPARK-21737
> URL: https://issues.apache.org/jira/browse/SPARK-21737
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Jong Yoon Lee
> Priority: Minor
>
> In this JIRA, I develop code to create a communication channel between
> arbitrary clients and a Spark AM on YARN. This code can be utilized to send
> commands such as getting status command, getting history info from the CLI,
> killing the application and pushing new tokens.
> Design Doc:
> https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
