[
https://issues.apache.org/jira/browse/SAMZA-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047905#comment-14047905
]
Martin Kleppmann commented on SAMZA-307:
----------------------------------------
I think this is a good idea — I also feel the deployment process is currently
hard for new users to understand. A few points/questions:
# Where would that deployment script live? We'd probably need to make a binary
release, so that users can download a Samza .tar.gz file, unpack it, and run
the deployment script directly. I've opened SAMZA-311 for that.
# The process involves several distinct steps (building a job package,
uploading it to HDFS, telling YARN to run the job). While it's useful to do all
three steps at once for projects that need it, it would be worth separating out
the stages and allowing users to run only some of them (e.g. skip the job
package building for projects that already have a suitable build process). It
might even make sense to make those steps separate JIRAs, because building a
job package is quite distinct from uploading a package to HDFS.
# How will the script choose the HDFS location to use?
# Will the job package in HDFS get cleaned up after a job is shut down?
# Can we support Hadoop clusters that have security enabled for HDFS?
> Simplify YARN deploy procedure
> -------------------------------
>
> Key: SAMZA-307
> URL: https://issues.apache.org/jira/browse/SAMZA-307
> Project: Samza
> Issue Type: Improvement
> Reporter: Yan Fang
>
> Currently, we have two ways of deploying the samza job to YARN cluster, from
> [HDFS|https://samza.incubator.apache.org/learn/tutorials/0.7.0/deploy-samza-job-from-hdfs.html]
> and [Http |
> https://samza.incubator.apache.org/learn/tutorials/0.7.0/run-in-multi-node-yarn.html],
> but neither of them is out-of-box. Users have to go through the tutorial,
> add dependencies, recompile, put the job package to HDFS or Http and then
> finally run. I feel it is a little cumbersome sometimes. We maybe able to
> provide a simpler way to deploy the job.
> When users have YARN and HDFS in the same cluster (such as CDH5), we can
> provide a job-submit script which does:
> 1. take cluster configuration
> 2. call some jave code to upload the assembly (all the samza needed jars and
> is already-compiled) and user's job jar (which changes frequently) to the HDFS
> 3. run the job as usual.
> Therefore, the users only need to run one command line *instead of*:
> 1. going step by step from the tutorial during their first job
> 2. assembling all code and uploading to HDFS manually every time they make
> changes to their job.
> (Yes, I learnt it from [Spark's Yarn
> deploy|http://spark.apache.org/docs/latest/running-on-yarn.html] and [their
> code|https://github.com/apache/spark/blob/master/yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala]
> )
> When users only have YARN, I think they have no way but start the http server
> as tutorial.
> What do you think? Does the simplification make sense? Or the Samza will have
> some difficulties (issues) if we do the deploy in this way? Thank you.
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
