[
https://issues.apache.org/jira/browse/SAMZA-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050308#comment-14050308
]
Zhijie Shen commented on SAMZA-307:
-----------------------------------
bq. What to Tez and other YARN frameworks do?
Yes, the location could be arbitrary for Tez. MR can be deployed in a similar
way.
bq. Can we support Hadoop clusters that have security enabled for HDFS?
Yes, we can. However, if I understand correctly, Samza job can still not run on
a secured YARN cluster, right? Then, the job may not able to localized the jars
from HDFS.
I'm +1 for separating Samza runtime jars and user jars.
* Samza runtime jars: we should deploy it once for a cluster. It is not aligned
withe any Samza job life cycle. All Samza jobs should be able to reuse it. We
can even deploy different versions of Samza runtime in a YARN cluster, and a
job can pick a certain version to use. For these jars, we can choose to put
them in some dir like /apps/samza/0.x.y/ by default.
* User jars: For a particular Samza job, the user jar that contains the job's
specific logic needs to be uploaded, localized the where it's going to be
executed, and finally be cleaned after the job is terminated.
> Simplify YARN deploy procedure
> -------------------------------
>
> Key: SAMZA-307
> URL: https://issues.apache.org/jira/browse/SAMZA-307
> Project: Samza
> Issue Type: Improvement
> Reporter: Yan Fang
>
> Currently, we have two ways of deploying the samza job to YARN cluster, from
> [HDFS|https://samza.incubator.apache.org/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html]
> and [Http |
> https://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-node-yarn.html],
> but neither of them is out-of-box. Users have to go through the tutorial,
> add dependencies, recompile, put the job package to HDFS or Http and then
> finally run. I feel it is a little cumbersome sometimes. We maybe able to
> provide a simpler way to deploy the job.
> When users have YARN and HDFS in the same cluster (such as CDH5), we can
> provide a job-submit script which does:
> 1. take cluster configuration
> 2. call some jave code to upload the assembly (all the samza needed jars and
> is already-compiled) and user's job jar (which changes frequently) to the HDFS
> 3. run the job as usual.
> Therefore, the users only need to run one command line *instead of*:
> 1. going step by step from the tutorial during their first job
> 2. assembling all code and uploading to HDFS manually every time they make
> changes to their job.
> (Yes, I learnt it from [Spark's Yarn
> deploy|http://spark.apache.org/docs/latest/running-on-yarn.html] and [their
> code|https://github.com/apache/spark/blob/master/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala]
> )
> When users only have YARN, I think they have no way but start the http server
> as tutorial.
> What do you think? Does the simplification make sense? Or the Samza will have
> some difficulties (issues) if we do the deploy in this way? Thank you.
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
