[
https://issues.apache.org/jira/browse/SAMZA-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288053#comment-14288053
]
Yi Pan (Data Infrastructure) commented on SAMZA-516:
----------------------------------------------------
[~criccomini], just some quick thoughts:
#Is the run-job.sh executed for each job on each node? Is it worth thinking of
running a daemon process that registers the available daemon processes to
_process_group_ on ZK, and then each daemon process watches a _jobs_ ZK
directory for new job node? Once a new job node is added, all daemon processes
can try to grab the ephemeral node for the job and the one acquires the
ephemeral node immediately becomes the leader (i.e. JobCoordinator)? The leader
will then pick and choose the processes that would be part of this job and do
the same container assignments as in Step 1.5
##One benefit of doing this is that new job creation will simply be a ZK write
to add the job node under jobs ZK directory. No need to run run-job.sh on each
node for each new job.
##The other benefit is that the JobCoordinator can immediately pick up multiple
registered processes for the job and do assignment once, instead of requiring
re-assign when each node runs run-job.sh (as in Step 2.4)
##Down-side: we need to use something similar to daemontools to run daemon
processes as a service, bringing in some external dependencies
{quote}
When we move the AM UI out of YARN and into JobCoordinator, you'll be able to
use the UI in standalone mode, but the UI will jump from node to node as
machines fail. This seems kind of annoying.
{quote}
How heavy is it to run a light-weighted web-service on each node keep tracking
of the JobCoordinator location and re-direct the request?
> Support standalone Samza jobs
> -----------------------------
>
> Key: SAMZA-516
> URL: https://issues.apache.org/jira/browse/SAMZA-516
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.9.0
> Reporter: Chris Riccomini
> Assignee: Chris Riccomini
>
> Samza currently supports two modes of operation out of the box: local and
> YARN. With local mode, a single Java process starts the JobCoordinator,
> creates a single container, and executes it locally. All partitions are
> procesed within this container. With YARN, a YARN grid is required to
> execute the Samza job. In addition, SAMZA-375 introduces a patch to run Samza
> in Mesos.
> There have been several requests lately to be able to run Samza jobs without
> any resource manager (YARN, Mesos, etc), but still run it in a distributed
> fashion.
> The goal of this ticket is to design and implement a samza-standalone module,
> which will:
> # Support executing a single Samza job in one or more containers.
> # Support failover, in cases where a machine is lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
