[
https://issues.apache.org/jira/browse/KYLIN-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960466#comment-14960466
]
liyang commented on KYLIN-1021:
-------------------------------
I've merge the patch in commit 88c660fd2b2a7aa6654b8929b831c6f861bcf51b
Note the final impl is slightly different from the original patch. Instead of
blindly upload all hbase/hive/hadoop dependencies, a configurable dir
"kylin.job.mr.lib.dir" is specified and all its content will be add to
"tmpjars" and "tmpfiles" of MR job.
> upload dependent jars of kylin to HDFS and set tmpjars
> ------------------------------------------------------
>
> Key: KYLIN-1021
> URL: https://issues.apache.org/jira/browse/KYLIN-1021
> Project: Kylin
> Issue Type: Improvement
> Affects Versions: v1.0
> Reporter: fengYu
> Assignee: liyang
> Attachments: 0001-hadoop-jar-files.patch
>
>
> As [~Shaofengshi] says in maillist : Regrading your question about the jar
> files located in local disk instead of HDFS, yes the hadoop/hive/hbase jars
> should exist in local disk on each machine of the hadoop cluster, with the
> same locations; Kylin will not upload those jars; Please check and ensure the
> consistency of your hadoop cluster.
> However, our hadoop cluster is managed by hadoop administrator, we have no
> right to login those machine, even though we have the right, copy all files
> to hundreds of machine will be a painful job(I do not know is there any tools
> can do the job well).
> By the way, I can not get any tips about you measure(If you has the document，
> tell me)...
> I change my source code and create a directory in kylin tmp
> directory(kylin.hdfs.working.dir/kylin_metadata) and upload all jars to the
> directory if the directory is empty(it only happened at the first time) when
> submitting a mapreduce job, then set those locations to tmpjars of the
> mapreduce job(just like kylin set tmpfiles before submit job), This is
> automated and make kylin deploying easier..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
