[
https://issues.apache.org/jira/browse/KYLIN-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743038#comment-14743038
]
liyang commented on KYLIN-1021:
-------------------------------
We have to first distinguish "platform" and "application", then can answer if
hive/hbase jars should be uploaded.
Platform is provided computation infra. Your Hadoop admin manages the platform
and as a user, you never upload a platform jar as part of MR job. For example,
HDFS / YARN / MR are obvious platform components. On the other hand,
application jar like kylin must be uploaded as part of MR job.
Now the question is HBase/Hive is platform or application? My take is they are
platform components in most Hadoop deployment. That's why I support the patch,
but it should be off by default.
> upload dependent jars of kylin to HDFS and set tmpjars
> ------------------------------------------------------
>
> Key: KYLIN-1021
> URL: https://issues.apache.org/jira/browse/KYLIN-1021
> Project: Kylin
> Issue Type: Improvement
> Affects Versions: v1.0
> Reporter: fengYu
>
> As [~Shaofengshi] says in maillist : Regrading your question about the jar
> files located in local disk instead of HDFS, yes the hadoop/hive/hbase jars
> should exist in local disk on each machine of the hadoop cluster, with the
> same locations; Kylin will not upload those jars; Please check and ensure the
> consistency of your hadoop cluster.
> However, our hadoop cluster is managed by hadoop administrator, we have no
> right to login those machine, even though we have the right, copy all files
> to hundreds of machine will be a painful job(I do not know is there any tools
> can do the job well).
> By the way, I can not get any tips about you measure(If you has the document，
> tell me)...
> I change my source code and create a directory in kylin tmp
> directory(kylin.hdfs.working.dir/kylin_metadata) and upload all jars to the
> directory if the directory is empty(it only happened at the first time) when
> submitting a mapreduce job, then set those locations to tmpjars of the
> mapreduce job(just like kylin set tmpfiles before submit job), This is
> automated and make kylin deploying easier..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
