[
https://issues.apache.org/jira/browse/GIRAPH-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Reisman updated GIRAPH-479:
-------------------------------
Attachment: GIRAPH-479-4.patch
Just a rebase. This is a nice refactor but may not be needed to make YARN work,
so its low priority until more of that work is done. This does work, and passes
mvm verify if anyone does see a compelling reason to move on it sooner.
It is very possible (even easy) when the YARN impl is up and running that we
can just have our ApplicationMaster launch an instance of ZK if the
giraph.zkList conf field is empty, and then populate that field with the host
and port of our new ZK instance to "mimic" the local instance we use on MRv1.
When the workers are instantiated, they will already have the correct ZK
address in their conf and won't even know if its an app-local or standalone
instance! Whenever the ZkManager gets fired up, it sees zkList is populated and
just goes back to sleep, allowing YARN to handle this part of the ZK function
without changing the MRv1 ZK code at all.
We could even delay the rest of the launches for a bit to guarantee its up and
running before we start our workers. So there's a number of things brewing
around this right now that might make it nice but not needed. But I digress...
> Factor creation of job-local ZooKeeper instances out of ZKManager to enable
> YARN instantiation, etc.
> ----------------------------------------------------------------------------------------------------
>
> Key: GIRAPH-479
> URL: https://issues.apache.org/jira/browse/GIRAPH-479
> Project: Giraph
> Issue Type: Improvement
> Components: zookeeper
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Attachments: GIRAPH-479-1.patch, GIRAPH-479-2.patch,
> GIRAPH-479-3.patch, GIRAPH-479-4.patch
>
>
> When the user does not specify a ZooKeeper quorum to run Giraph on, the
> ZooKeeperManager creates one. This factors out the boilerplate that handles
> this additional process and its lifecycle so that various pluggable versions
> of this process instantiation can be implemented and fed to the ZKManager via
> a factory depending on the nature of the underlying cluster.
> By implementing the ZooKeeperProcessBuilder interface, one will now be able
> to implement the creation/mangement of a new, job-local ZK instance using any
> underlying resourcing mechanism one chooses (including, I hope, YARN.)
> Passes mvn verify, etc.
> Review board link:
> https://reviews.apache.org/r/8948/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
