[jira] [Commented] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[ https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589509#comment-14589509 ] Hari Sekhon commented on MAPREDUCE-6401: Yes the attached log is from the container task logs obtained via the RM. I would have expected a failing shell call should print the command, status code and both stderr and stdout with which to debug/reproduce. It's clear something is different since these are new nodes in the cluster that are giving these problems but it's not clear what is wrong because the debug output is so vague - I'm trying to fix it blindfolded. Container-launch failure gives no debugging output -- Key: MAPREDUCE-6401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Attachments: job.log MR jobs are failing on my cluster with Stack trace: ExitCodeException exitCode=7 but little else in terms of debugging information. Can we please improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[
https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589925#comment-14589925
]
Hari Sekhon commented on MAPREDUCE-6401:
Actually the task logs showed the same thing, not much to go on:
{code}
Exception from container-launch. Container id:
container_e199_1434474871820_0001_02_19 Exit code: 7 Stack trace:
ExitCodeException exitCode=7:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:293)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) Shell output: main : command
provided 1 main : user is custom_scrubbed main : requested yarn user is
custom_scrubbed Container exited with a non-zero exit code 7
{code}
but the full tasks logs don't seem to have been retained by the history server.
This made me suspicious so I reset the logging locations to try to get my hands
on the full logs and after a yarn restart jobs started working normally again
without failed tasks/container launches. Although I'm very certain that the
cluster used to log to that dir I reset it to, perhaps Ambari had a bug that
lost the location and reset to debug locations that didn't work properly (it
wouldn't be the first time, eg. AMBARI-9022)
I think we should leave this as a minor todo to improve debugging information,
especially when launching shell commands and encountering non-zero exit codes,
logging is king.
Container-launch failure gives no debugging output
--
Key: MAPREDUCE-6401
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Environment: HDP 2.2
Reporter: Hari Sekhon
Attachments: job.log
MR jobs are failing on my cluster with Stack trace: ExitCodeException
exitCode=7 but little else in terms of debugging information. Can we please
improve the debugging info? Log file is attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[ https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6401: --- Priority: Minor (was: Major) Container-launch failure gives no debugging output -- Key: MAPREDUCE-6401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Priority: Minor Attachments: job.log MR jobs are failing on my cluster with Stack trace: ExitCodeException exitCode=7 but little else in terms of debugging information. Can we please improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6401) Container-launch failure gives no debugging output
Hari Sekhon created MAPREDUCE-6401: -- Summary: Container-launch failure gives no debugging output Key: MAPREDUCE-6401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Attachments: job.log MR jobs are failing on my cluster with Stack trace: ExitCodeException exitCode=7 but little else in terms of debugging information. Can we please improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6401) Container-launch failure gives no debugging output
[ https://issues.apache.org/jira/browse/MAPREDUCE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6401: --- Attachment: job.log Container-launch failure gives no debugging output -- Key: MAPREDUCE-6401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6401 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.6.0 Environment: HDP 2.2 Reporter: Hari Sekhon Attachments: job.log MR jobs are failing on my cluster with Stack trace: ExitCodeException exitCode=7 but little else in terms of debugging information. Can we please improve the debugging info? Log file is attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-210) want InputFormat for zip files
[ https://issues.apache.org/jira/browse/MAPREDUCE-210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356797#comment-14356797 ] Hari Sekhon commented on MAPREDUCE-210: --- There is 3rd party zip inputformat here: http://cotdp.com/2012/07/hadoop-processing-zip-files-in-mapreduce/ I think it's important for the zip inputformat to be natively supported because traditional enterprises where Hadoop is now starting to penetrate use zip a lot, especially in large corporates which are Windows heavy and don't realize the problems they are causing by having so many things in zip files that Hadoop currently can't read. want InputFormat for zip files -- Key: MAPREDUCE-210 URL: https://issues.apache.org/jira/browse/MAPREDUCE-210 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Doug Cutting Assignee: indrajit Attachments: ZipInputFormat_fixed.patch HDFS is inefficient with large numbers of small files. Thus one might pack many small files into large, compressed, archives. But, for efficient map-reduce operation, it is desireable to be able to split inputs into smaller chunks, with one or more small original file per split. The zip format, unlike tar, permits enumeration of files in the archive without scanning the entire archive. Thus a zip InputFormat could efficiently permit splitting large archives into splits that contain one or more archived files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6270) Doc typo
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6270: --- Issue Type: Bug (was: Task) Doc typo Key: MAPREDUCE-6270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Reporter: Hari Sekhon Assignee: Hari Sekhon Priority: Trivial Disregard, wrong project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6270) Doc typo Mime Magic Detction
Hari Sekhon created MAPREDUCE-6270: -- Summary: Doc typo Mime Magic Detction Key: MAPREDUCE-6270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Reporter: Hari Sekhon Priority: Trivial On this page http://tika.apache.org/1.7/detection.html it says in one heading Mime Magic Detction instead of Mime Magic Detection. I don't know if that's an intentional joke :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6270) Doc typo Mime Magic Detction
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon resolved MAPREDUCE-6270. Resolution: Invalid Sorry raised this against the wrong project. Doc typo Mime Magic Detction Key: MAPREDUCE-6270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Reporter: Hari Sekhon Priority: Trivial On this page http://tika.apache.org/1.7/detection.html it says in one heading Mime Magic Detction instead of Mime Magic Detection. I don't know if that's an intentional joke :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6270) Doc typo Mime Magic Detction
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6270: --- Description: Disregard, wrong project. (was: On this page http://tika.apache.org/1.7/detection.html it says in one heading Mime Magic Detction instead of Mime Magic Detection. I don't know if that's an intentional joke :)) Doc typo Mime Magic Detction Key: MAPREDUCE-6270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Reporter: Hari Sekhon Assignee: Hari Sekhon Priority: Trivial Disregard, wrong project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6270) Doc typo
[ https://issues.apache.org/jira/browse/MAPREDUCE-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated MAPREDUCE-6270: --- Summary: Doc typo (was: Doc typo Mime Magic Detction) Doc typo Key: MAPREDUCE-6270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6270 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Reporter: Hari Sekhon Assignee: Hari Sekhon Priority: Trivial Disregard, wrong project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
