[
https://issues.apache.org/jira/browse/SAMZA-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028388#comment-14028388
]
Chris Riccomini commented on SAMZA-37:
--------------------------------------
[note] I'm using run-container.sh instead of run-task.sh in this comment
because the file has been renamed.
bq. Is it a typo? yarm.am.execute=bin/run-task.sh ->
yarm.am.execute=bin/run-am.sh ?
Correct, sorry, that was a typo.
bq. What does "disable run-task.sh override" mean? Is it not supported now or
will we plan to not support it?
Currently, when the Samza YARN AM starts a new YARN container, it tells YARN to
execute a command to start the container. This command defaults to
bin/run-container.sh. Samza allows developers to override this command using
the task.execute and task.command.class commands. The use-case for this is to
allow people to run a custom SamzaContainer implementation (in another
language), not just the Java-based version. The thought was that people might
want to have a bin/my-python-task script, or something.
I'm not quite sure that we want to touch this right now. To some extent, it's
dependent on how the multi-language support ends up looking (C library vs.
inside SamzaContainer with STDIN/STDOUT pipes). If you have a thought on how we
should handle the run-task.sh override settings, I'm happy to hear it, but I
don't really have much opinion at this point.
> Allow run-am.sh override for YARN AM or disable run-task.sh override
> --------------------------------------------------------------------
>
> Key: SAMZA-37
> URL: https://issues.apache.org/jira/browse/SAMZA-37
> Project: Samza
> Issue Type: Bug
> Components: container, yarn
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
>
> Samza allows developers to over-ride the path for the run-task.sh script, but
> does not allow developers to override the path for the run-am.sh script. The
> run-am.sh script is hard coded to __package/bin/run-am.sh.
> The two relevant configurations that allow run-task.sh override are:
> * task.command.class=org.apache.samza.job.ShellCommandBuilder
> * task.execute=bin/run-task.sh
> The motivation for allowing this override is to support Samza job packages
> with different package layouts, and also to allow for supporting other
> languages (e.g. running a python virtual environment as a Samza job).
> If we believe that the run-task.sh override feature is a good thing (one
> might argue it's kind of complex, and we should just force a convention),
> then we should allow run-am.sh override using the same style. If we go this
> route, we should add the following configuration to YarnConfig:
> * yarn.am.command.class=org.apache.samza.job.YarnAmShellCommandBuilder
> * yarm.am.execute=bin/run-task.sh
> The YarnAmShellCommandBuilder should just extend ShellCommandBuilder, but
> override buildCommand to call config.getAmCommand (yarn.am.execute), which
> will be defined in YarnConfig.
> I think we should just force the bin/run-task.sh convention, and disable
> run-task.sh override. We can still keep the ShellCommand* classes around,
> since they help build the environment variables for the SamzaContainer.
> I also think we should rename run-task.sh to run-container.sh, but I'll open
> a separate ticket for that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
