[ https://issues.apache.org/jira/browse/SAMZA-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181405#comment-15181405 ]
Yi Pan (Data Infrastructure) commented on SAMZA-881: ---------------------------------------------------- Hi, [~jagadish1...@gmail.com], the proposal looks good to me. I am attaching a few comments below: # When we describe mutual exclusiveness in partition assignment, please exclude broadcast stream in the discussion # Be consistent w/ terms: in page 2, “leader container” vs “leader process” # In the architecture graph, it would be nice to label the text on the edges w/ execution order # Where is the container liveness management module in the design of JobCoordinator? W/ SAMZA-871, requesting for direct heart beat between the containers and AM (i.e. essentially followers and leaders in the new design), I think that we should have a separate pluggable module for this, in addition to ContainerProcessManager, which is just interface to allocate/request processes. # Case 3.2 is the prototype implemented in SAMZA-516, right? We should call it out. Thanks a lot for the design! > Re-think the Samza Job Coordinator > ---------------------------------- > > Key: SAMZA-881 > URL: https://issues.apache.org/jira/browse/SAMZA-881 > Project: Samza > Issue Type: Improvement > Reporter: Jagadish > Assignee: Jagadish > Attachments: SamzaJobCoordinatorRe-designProposal.pdf > > > Currently, the only way to run Samza containers in distributed mode is using > Yarn. However, there has been interest to run Samza on top of other resource > managers with the recent explosion in the # of such systems. Users have also > requested us to run Samza as a library, and to run Samza on Docker containers > managed by Kubernetes. > We must re-think the JobCoordinator functionality as follows: > 1. ID assignment: Provide an ID to each SamzaContainer. > 2. JobModel agreement: Ensure containers agree on a JobModel. > 3. Re-start the SamzaContainer when the job model changes. > This will arguably require some leader election (depending on how users > choose to run Samza). -- This message was sent by Atlassian JIRA (v6.3.4#6332)