[
https://issues.apache.org/jira/browse/KYLIN-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-629.
------------------------------
> Kylin failed to run mapreduce job if there is no
> mapreduce.application.classpath in mapred-site.xml
> ---------------------------------------------------------------------------------------------------
>
> Key: KYLIN-629
> URL: https://issues.apache.org/jira/browse/KYLIN-629
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v0.7.1
> Reporter: Shaofeng SHI
> Assignee: Shaofeng SHI
> Priority: Major
> Fix For: v0.7.1
>
>
> I deployed 0.7.1 snapshot in our hadoop cluster, sync a table to Kylin, which
> trigger Kylin submit a hadoop job to calculate the fields cardinality; But
> this job was failed, the error is:
> 15/03/10 01:05:25 INFO mapreduce.Job: Job job_1425075571333_139544 failed
> with state FAILED due to: Application application_1425075571333_139544 failed
> 2 times due to AM Container for appattempt_1425075571333_139544_000002 exited
> with exitCode: 1 due to: Exception from container-launch:
> org.apache.hadoop.util.Shell$ExitCodeException:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
> at org.apache.hadoop.util.Shell.run(Shell.java:418)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Adding more log in AbstractHadoopJob.setJobClasspath, get the following debug
> message:
> [pool-7-thread-1]:[2015-03-10
> 00:33:17,739][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:146)]
> - append job jar:
> /export/home/b_kylin/kylin_ii/lib/kylin-job-0.7.1-SNAPSHOT.jar
> [pool-7-thread-1]:[2015-03-10
> 00:33:17,740][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:152)]
> - append kylin.hive.dependency:
> /apache/hive/conf:/apache/hive/lib/*:/apache/hive-0.13.0.2.1.3.0-563/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.2.1.3.0-563.jar
> to mapreduce.application.classpath
> [pool-7-thread-1]:[2015-03-10
> 00:33:17,740][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:164)]
> - Hadoop job classpath is:
> /apache/hive/conf,/apache/hive/lib/*,/apache/hive-0.13.0.2.1.3.0-563/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.2.1.3.0-563.jar
> From it I see the job configuration's "mapreduce.application.classpath" was
> empty before appending the hive dependencies; After appending, there is only
> hive on the classpath, which would cause the hadoop job failed to run.
> Doing some search, find the explaination of "mapreduce.application.classpath"
> in:
> https://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
> Now we know that, the added hive dependency will overwrite the default
> classpath, which is not expected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
