Eli Reisman created GIRAPH-574:
----------------------------------
Summary: Move Giraph Master node functionality to AppMaster or
launch directly from AppMaster in YARN profile
Key: GIRAPH-574
URL: https://issues.apache.org/jira/browse/GIRAPH-574
Project: Giraph
Issue Type: Improvement
Reporter: Eli Reisman
Priority: Minor
As folks read the Giraph on YARN code it is inevitable it will occur to someone
"Well, if the job fails when the ApplicationMaster fails, could we move some or
all of our Master task functions there and just call it master?"
Yes. In two ways.
One, we launch a dedicated master process marked as such with setup
responsibilities, and we assess from the app master how the launch went. We
keep launching "masters" until one takes. Then, we launch the workers.
Another is to simply run MasterThread and associated stuff from the App Master
directly, and when we know its up and running properly, only then does app
master launch the workers.
The YARN app master can be rebooted is designed to be a place for
fault-tolerant "master node" stuff to happen. However, I think a larger purpose
is to act as a meta-master for launching a DAG of jobs within the run of a
single app master lifecycle. Or the app master cant act as any of these things,
or something else I haven't thought of. The architecture is fairly malleable.
This is not a requirement for us, and maybe not a good idea at all. This is
just a placeholder JIRA to discuss and collect ideas since as I said above
someone is going to bring it up ;)
Thank you for reading.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
