[jira] [Commented] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[
https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589925#comment-14589925
]
Hari Sekhon commented on MAPREDUCE-6401:
Actually the task logs showed the same thing, not much to go on:
{code}
Exception from container-launch. Container id:
container_e199_1434474871820_0001_02_19 Exit code: 7 Stack trace:
ExitCodeException exitCode=7:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) Shell output: main : command
provided 1 main : user is main : requested yarn user is
Container exited with a non-zero exit code 7
{code}
but the full tasks logs don't seem to have been retained by the history server.
This made me suspicious so I reset the logging locations to try to get my hands
on the full logs and after a yarn restart jobs started working normally again
without failed tasks/container launches. Although I'm very certain that the
cluster used to log to that dir I reset it to, perhaps Ambari had a bug that
lost the location and reset to debug locations that didn't work properly (it
wouldn't be the first time, eg. AMBARI-9022)
I think we should leave this as a minor todo to improve debugging information,
especially when launching shell commands and encountering non-zero exit codes,
logging is king.
> Container-launch failure gives no debugging output
> --
>
> Key: MAPREDUCE-6401
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.6.0
> Environment: HDP 2.2
>Reporter: Hari Sekhon
> Attachments: job.log
>
>
> MR jobs are failing on my cluster with Stack trace: ExitCodeException
> exitCode=7 but little else in terms of debugging information. Can we please
> improve the debugging info? Log file is attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[ https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6401: --- Priority: Minor (was: Major) > Container-launch failure gives no debugging output > -- > > Key: MAPREDUCE-6401 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.6.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon >Priority: Minor > Attachments: job.log > > > MR jobs are failing on my cluster with Stack trace: ExitCodeException > exitCode=7 but little else in terms of debugging information. Can we please > improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[ https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589509#comment-14589509 ] Hari Sekhon commented on MAPREDUCE-6401: Yes the attached log is from the container task logs obtained via the RM. I would have expected a failing shell call should print the command, status code and both stderr and stdout with which to debug/reproduce. It's clear something is different since these are new nodes in the cluster that are giving these problems but it's not clear what is wrong because the debug output is so vague - I'm trying to fix it blindfolded. > Container-launch failure gives no debugging output > -- > > Key: MAPREDUCE-6401 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.6.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > Attachments: job.log > > > MR jobs are failing on my cluster with Stack trace: ExitCodeException > exitCode=7 but little else in terms of debugging information. Can we please > improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[ https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6401: --- Attachment: job.log > Container-launch failure gives no debugging output > -- > > Key: MAPREDUCE-6401 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.6.0 > Environment: HDP 2.2 >Reporter: Hari Sekhon > Attachments: job.log > > > MR jobs are failing on my cluster with Stack trace: ExitCodeException > exitCode=7 but little else in terms of debugging information. Can we please > improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6401) Container-launch failure gives no debugging output
Hari Sekhon created MAPREDUCE-6401: -- Summary: Container-launch failure gives no debugging output Key: MAPREDUCE-6401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Attachments: job.log MR jobs are failing on my cluster with Stack trace: ExitCodeException exitCode=7 but little else in terms of debugging information. Can we please improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-210) want InputFormat for zip files
[ https://issues.apache.org/jira/browse/MAPREDUCE-210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356797#comment-14356797 ] Hari Sekhon commented on MAPREDUCE-210: --- There is 3rd party zip inputformat here: http://cotdp.com/2012/07/hadoop-processing-zip-files-in-mapreduce/ I think it's important for the zip inputformat to be natively supported because traditional enterprises where Hadoop is now starting to penetrate use zip a lot, especially in large corporates which are Windows heavy and don't realize the problems they are causing by having so many things in zip files that Hadoop currently can't read. > want InputFormat for zip files > -- > > Key: MAPREDUCE-210 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-210 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Doug Cutting >Assignee: indrajit > Attachments: ZipInputFormat_fixed.patch > > > HDFS is inefficient with large numbers of small files. Thus one might pack > many small files into large, compressed, archives. But, for efficient > map-reduce operation, it is desireable to be able to split inputs into > smaller chunks, with one or more small original file per split. The zip > format, unlike tar, permits enumeration of files in the archive without > scanning the entire archive. Thus a zip InputFormat could efficiently permit > splitting large archives into splits that contain one or more archived files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6270) Doc typo
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6270: --- Issue Type: Bug (was: Task) > Doc typo > > > Key: MAPREDUCE-6270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Hari Sekhon >Assignee: Hari Sekhon >Priority: Trivial > > Disregard, wrong project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6270) Doc typo Mime Magic Detction
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon reassigned MAPREDUCE-6270: -- Assignee: Hari Sekhon > Doc typo Mime Magic Detction > > > Key: MAPREDUCE-6270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: documentation >Reporter: Hari Sekhon >Assignee: Hari Sekhon >Priority: Trivial > > On this page http://tika.apache.org/1.7/detection.html it says in one heading > "Mime Magic Detction" instead of "Mime Magic Detection". I don't know if > that's an intentional joke :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6270) Doc typo Mime Magic Detction
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6270: --- Description: Disregard, wrong project. (was: On this page http://tika.apache.org/1.7/detection.html it says in one heading "Mime Magic Detction" instead of "Mime Magic Detection". I don't know if that's an intentional joke :)) > Doc typo Mime Magic Detction > > > Key: MAPREDUCE-6270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: documentation >Reporter: Hari Sekhon >Assignee: Hari Sekhon >Priority: Trivial > > Disregard, wrong project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6270) Doc typo
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6270: --- Summary: Doc typo (was: Doc typo Mime Magic Detction) > Doc typo > > > Key: MAPREDUCE-6270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: documentation >Reporter: Hari Sekhon >Assignee: Hari Sekhon >Priority: Trivial > > Disregard, wrong project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6270) Doc typo Mime Magic Detction
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon resolved MAPREDUCE-6270. Resolution: Invalid Sorry raised this against the wrong project. > Doc typo Mime Magic Detction > > > Key: MAPREDUCE-6270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: documentation >Reporter: Hari Sekhon >Priority: Trivial > > On this page http://tika.apache.org/1.7/detection.html it says in one heading > "Mime Magic Detction" instead of "Mime Magic Detection". I don't know if > that's an intentional joke :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6270) Doc typo Mime Magic Detction
Hari Sekhon created MAPREDUCE-6270: -- Summary: Doc typo Mime Magic Detction Key: MAPREDUCE-6270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Reporter: Hari Sekhon Priority: Trivial On this page http://tika.apache.org/1.7/detection.html it says in one heading "Mime Magic Detction" instead of "Mime Magic Detection". I don't know if that's an intentional joke :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
