[
https://issues.apache.org/jira/browse/SAMZA-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110284#comment-14110284
]
Jon Bringhurst commented on SAMZA-375:
--------------------------------------
{quote}
One question regarding the screencap. I'm not terribly familiar with Mesos.
Based on the UI, it looks like it's looking at each Samza job as an independent
"framework". Is this the suggested way to integrate Samza with Mesos, or do we
need to provide some kind of scheduler, which Mesos sees just one of, and is
"the framework"? I could see us wanting to move the YARN AM logic to a single
centralized UI in Mesos, which plugs into Mesos. This UI/service could then
manage all jobs.
I'm just not too clear on what the standard pattern is here.
{quote}
Hey [~criccomini], here's what I have in mind: Each Samza job would run as a
framework which manages tasks within a single job (the scheduler). Then,
another "meta-framework" (a phrase that mesosphere seems to be using) would run
to manage each of the frameworks for the Samza jobs. Initially, the
meta-framework would be something like Marathon or Aurora, but could later
evolve into a Samza-specific meta-framework, perhaps with a special interface
for managing multiple Samza jobs and streams. Also, a Samza-specific
meta-framework may be a useful place to download project packages from, or at
least coordinate where the packages may be located. In the far future,
something like Marathon's eventbus may be useful for communicating with Kafka
(or other applications).
Another alternative would be to have a scheduler for multiple jobs. However, it
seems like this would require large changes to Samza (the MesosJobLauncher
concept wouldn't really fit well) and would reduce fault tolerance since it
would concentrate complex logic within a single process. However, it wouldn't
reduce fault tolerance that much, so I'm definitely open to the idea. Perhaps
the MesosJobLauncher could make an RPC call out to a Samza Mesos scheduler,
which would then coordinate resources for the new job (I'm just brainstorming
here, this specific idea seems kludgy).
{quote}
Also, regarding this log line: "Received offer id ...", could we make it a
single-line log? Not too excited about multi-line logs.
{quote}
Agreed. This was just for debugging purposes.
> Investigate Mesos Job Support
> -----------------------------
>
> Key: SAMZA-375
> URL: https://issues.apache.org/jira/browse/SAMZA-375
> Project: Samza
> Issue Type: Bug
> Components: hello-samza
> Reporter: Jon Bringhurst
> Assignee: Jon Bringhurst
> Labels: mesos
> Attachments: Screen Shot 2014-08-23 at 5.51.39 PM.png
>
>
> It would be nice if Samza had support for Mesos (https://mesos.apache.org/).
> The current plan is to create a MesosJob and MesosJobFactory, then look into
> what it would take to allow the AM code to act as a Mesos scheduler.
> The feasibility of this landing in trunk will be better understood after a
> rough prototype has been created.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
