[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187501#comment-13187501 ] Hudson commented on MAPREDUCE-2450: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1569 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1569/]) MAPREDUCE-2450. Fixed a corner case with interrupted communication threads leading to a long timeout in Task. Contributed by Rajesh Balamohan. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232314 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Fix For: 1.0.0 > > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.apache.hadoop.ipc.Server.channelWrite(Server.java:11
[jira] [Resolved] (MAPREDUCE-3605) Allow mr commands to be run via bin/hadoop
[
https://issues.apache.org/jira/browse/MAPREDUCE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-3605.
--
Resolution: Duplicate
Assignee: (was: Prashant Sharma)
Duplicate of HADOOP-7971.
> Allow mr commands to be run via bin/hadoop
> --
>
> Key: MAPREDUCE-3605
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3605
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Ramya Sunil
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3605.patch
>
>
> MR command line options are not supported in bin/hadoop.
> {noformat}
> bin/hadoop job
> Exception in thread "main" java.lang.NoClassDefFoundError: job
> Caused by: java.lang.ClassNotFoundException: job
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: job. Program will exit.
> {noformat}
> A deprecated message like "DEPRECATED: Use of this script to execute mapred
> command is deprecated. Instead use the mapred command for it." should be
> displayed along with the correct output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187489#comment-13187489 ] Hudson commented on MAPREDUCE-2450: --- Integrated in Hadoop-Mapreduce-0.23-Commit #398 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/398/]) Merge -c 1232314 from trunk to branch-0.23 to fix MAPREDUCE-2450. MAPREDUCE-2450. Fixed a corner case with interrupted communication threads leading to a long timeout in Task. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232315 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Fix For: 1.0.0 > > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at
[jira] [Commented] (MAPREDUCE-3678) The Map tasks logs should have the value of input split it processed
[ https://issues.apache.org/jira/browse/MAPREDUCE-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187487#comment-13187487 ] Bejoy KS commented on MAPREDUCE-3678: - Ya it is available in taskdetails.jsp . But when we have a large number of jobs running on our cluster in a matter of half an hour the jobs would be in history and in in jobtaskshistory.jsp there are only the following values -Task Id -Start Time -Finish Time -Error Can we have one more filed here similar to status in taskdetails.jsp that would show the input split it processed as well. Once the job is in history viewer currently do we have any option to find this information? > The Map tasks logs should have the value of input split it processed > > > Key: MAPREDUCE-3678 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3678 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: nodemanager, tasktracker >Affects Versions: 0.20.203.0, 0.20.205.0, 1.0.0 > Environment: Linux red hat. >Reporter: Bejoy KS > > It would be easier to debug some corner in tasks if we knew what was the > input split processed by that task. Map reduce task tracker log should > accommodate the same. Also in the jobdetails web UI, the split also should be > displayed along with the Split Locations. > Sample as > Input Split > hdfs://myserver:9000/userdata/sampleapp/inputdir/file1.csv - no>/ > This would be much beneficial to nail down some data quality issues in large > data volume processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187485#comment-13187485 ] Hudson commented on MAPREDUCE-2450: --- Integrated in Hadoop-Common-trunk-Commit #1551 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1551/]) MAPREDUCE-2450. Fixed a corner case with interrupted communication threads leading to a long timeout in Task. Contributed by Rajesh Balamohan. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232314 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Fix For: 1.0.0 > > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.apache.hadoop.ipc.Server.channelWrite(Server.java:1195) >
[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187482#comment-13187482 ] Hudson commented on MAPREDUCE-2450: --- Integrated in Hadoop-Common-0.23-Commit #385 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/385/]) Merge -c 1232314 from trunk to branch-0.23 to fix MAPREDUCE-2450. MAPREDUCE-2450. Fixed a corner case with interrupted communication threads leading to a long timeout in Task. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232315 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Fix For: 1.0.0 > > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > or
[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187480#comment-13187480 ] Hudson commented on MAPREDUCE-2450: --- Integrated in Hadoop-Hdfs-trunk-Commit #1624 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1624/]) MAPREDUCE-2450. Fixed a corner case with interrupted communication threads leading to a long timeout in Task. Contributed by Rajesh Balamohan. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232314 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Fix For: 1.0.0 > > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.apache.hadoop.ipc.Server.channelWrite(Server.java:1195) >
[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187479#comment-13187479 ] Hudson commented on MAPREDUCE-2450: --- Integrated in Hadoop-Hdfs-0.23-Commit #375 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/375/]) Merge -c 1232314 from trunk to branch-0.23 to fix MAPREDUCE-2450. MAPREDUCE-2450. Fixed a corner case with interrupted communication threads leading to a long timeout in Task. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232315 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Fix For: 1.0.0 > > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.ap
[jira] [Updated] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2450: - Resolution: Fixed Fix Version/s: 1.0.0 Target Version/s: (was: 0.23.1) Status: Resolved (was: Patch Available) I just committed this to trunk & branch-0.23, already in branch-1. Thanks Rajesh! > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Fix For: 1.0.0 > > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.apache.hadoop.ipc.Server.channelWrite(Server.java:1195) > at org.apache.hadoop.ipc.Server.access$1900(Server.java:77) > at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:613) > at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:677) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:981) > > Note that the task actually seemed to commit - it didn't get speculatively > executed or a
[jira] [Commented] (MAPREDUCE-3675) A job must not be submitted when there are no mappers to run
[ https://issues.apache.org/jira/browse/MAPREDUCE-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187454#comment-13187454 ] Harsh J commented on MAPREDUCE-3675: Resolution is cool by me. I'll refer this ticket to folks requesting explanations henceforth - thanks for your specific comments Vinod and Arun! :) > A job must not be submitted when there are no mappers to run > > > Key: MAPREDUCE-3675 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3675 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 1.0.0 >Reporter: Harsh J >Priority: Minor > > Right now, one's able to run a job with an empty input directory and a job is > indeed scheduled. > The job runs no mappers, but any number of specified reducers are run anyway > as dummy tasks. > This should be/could be avoided. I do not see a use for such an allowance, > and it looks mostly like a logic slip to me with empty arrays involved and > accepted. > The fix could be simply in the job submission code, where we can avoid > submitting if the splits are nil. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187428#comment-13187428 ] Hadoop QA commented on MAPREDUCE-2450: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510796/MAPREDUCE-2450.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1617//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1617//console This message is automatically generated. > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apach
[jira] [Commented] (MAPREDUCE-3678) The Map tasks logs should have the value of input split it processed
[ https://issues.apache.org/jira/browse/MAPREDUCE-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187427#comment-13187427 ] Arun C Murthy commented on MAPREDUCE-3678: -- AFAIK MR1 already shows this in taskdetails.jsp - we need to add this to MR2. Also, AFAIK, I thought MR1 task-logs had this info logged, something I see missing in MR2 also. > The Map tasks logs should have the value of input split it processed > > > Key: MAPREDUCE-3678 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3678 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: nodemanager, tasktracker >Affects Versions: 0.20.203.0, 0.20.205.0, 1.0.0 > Environment: Linux red hat. >Reporter: Bejoy KS > > It would be easier to debug some corner in tasks if we knew what was the > input split processed by that task. Map reduce task tracker log should > accommodate the same. Also in the jobdetails web UI, the split also should be > displayed along with the Split Locations. > Sample as > Input Split > hdfs://myserver:9000/userdata/sampleapp/inputdir/file1.csv - no>/ > This would be much beneficial to nail down some data quality issues in large > data volume processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-199) Locality hints for Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-199: - Assignee: (was: Owen O'Malley) This should be much more feasible with MR2 and should yield significant benefits for small jobs. > Locality hints for Reduce > - > > Key: MAPREDUCE-199 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-199 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: applicationmaster, mrv2 >Reporter: Benjamin Reed > > It would be nice if we could add method to OutputFormat that would allow a > job to indicate where a reducer for a given partition should should run. This > is similar to the getSplits() method on InputFormat. In our application the > reducer is using other data in addition to the map outputs during processing > and data accesses could be made more efficient if the JobTracker scheduled > the reducers to run on specific hosts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-199) Locality hints for Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-199: Component/s: mrv2 applicationmaster > Locality hints for Reduce > - > > Key: MAPREDUCE-199 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-199 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: applicationmaster, mrv2 >Reporter: Benjamin Reed > > It would be nice if we could add method to OutputFormat that would allow a > job to indicate where a reducer for a given partition should should run. This > is similar to the getSplits() method on InputFormat. In our application the > reducer is using other data in addition to the map outputs during processing > and data accesses could be made more efficient if the JobTracker scheduled > the reducers to run on specific hosts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-460) Should be able to re-run jobs, collecting only missing output
[ https://issues.apache.org/jira/browse/MAPREDUCE-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-460: - Assignee: (was: Owen O'Malley) This is easy to support in MR2, I think we should take a crack at this - sounds like a very useful feature. > Should be able to re-run jobs, collecting only missing output > - > > Key: MAPREDUCE-460 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-460 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Bryan Pendleton >Priority: Minor > > For jobs with no side effects (roughly == jobs with speculative execution > enabled), if partial output has been generated, it should be possible to > re-run the job, and fill in the missing pieces. I have now run the same job > twice, once finishing 42 of 44 reduce tasks, another time finishing only 17. > Each time, many nodes have failed, causing many many tasks to fail ( in one > case, 5k failures from 15k map tasks, 23 failures from 44 reduces), but some > valid output was generated. Since the output is only dependent on the input, > and both jobs used the same input, I will now be able to combine these two > failed task outputs to get a completed job's output. This should be something > that can be more automatic. > In particular, it should be possible to resubmit a job, with a list of > partitions that should be ignored. A special Combiner, or pre-Combiner, would > throw out any map output for partitions that have already been successfully > completed, thus reducing the amount of data that needs to be reduced to > complete the job. It would, of course, be nice to support "filling in" > existing outputs, rather than having to do a move operation on completed > outputs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-460) Should be able to re-run jobs, collecting only missing output
[ https://issues.apache.org/jira/browse/MAPREDUCE-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-460: Component/s: mrv2 Priority: Major (was: Minor) > Should be able to re-run jobs, collecting only missing output > - > > Key: MAPREDUCE-460 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-460 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Bryan Pendleton > > For jobs with no side effects (roughly == jobs with speculative execution > enabled), if partial output has been generated, it should be possible to > re-run the job, and fill in the missing pieces. I have now run the same job > twice, once finishing 42 of 44 reduce tasks, another time finishing only 17. > Each time, many nodes have failed, causing many many tasks to fail ( in one > case, 5k failures from 15k map tasks, 23 failures from 44 reduces), but some > valid output was generated. Since the output is only dependent on the input, > and both jobs used the same input, I will now be able to combine these two > failed task outputs to get a completed job's output. This should be something > that can be more automatic. > In particular, it should be possible to resubmit a job, with a list of > partitions that should be ignored. A special Combiner, or pre-Combiner, would > throw out any map output for partitions that have already been successfully > completed, thus reducing the amount of data that needs to be reduced to > complete the job. It would, of course, be nice to support "filling in" > existing outputs, rather than having to do a move operation on completed > outputs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-218) Map/Reduce job with SequenceFileOutputFormat should be able to add user specified metadata to the output file
[ https://issues.apache.org/jira/browse/MAPREDUCE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-218. - Resolution: Fixed SequenceFiles already have a metadata sections. > Map/Reduce job with SequenceFileOutputFormat should be able to add user > specified metadata to the output file > - > > Key: MAPREDUCE-218 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-218 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Runping Qi > > When creating a map/reduce job with SequenceFileFormat, > the user would like to add some metada to the output files automatically. > In particular, if the output value class was a JuteRecord class generated > from a Jute IDL, > we would like to add JUTE_IDL/IDL_STRING as a attribute/value pair of the > metadata. > This way, the output files will be self describing: > When an application that tries to use the files may not have the value class > with it. > But the application can use Jute tool to generate the classes on demand. > Or better yet, the SequenceFile record reader may be able to do that > automatically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-218) Map/Reduce job with SequenceFileOutputFormat should be able to add user specified metadata to the output file
[ https://issues.apache.org/jira/browse/MAPREDUCE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-218: - > Map/Reduce job with SequenceFileOutputFormat should be able to add user > specified metadata to the output file > - > > Key: MAPREDUCE-218 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-218 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Runping Qi > > When creating a map/reduce job with SequenceFileFormat, > the user would like to add some metada to the output files automatically. > In particular, if the output value class was a JuteRecord class generated > from a Jute IDL, > we would like to add JUTE_IDL/IDL_STRING as a attribute/value pair of the > metadata. > This way, the output files will be self describing: > When an application that tries to use the files may not have the value class > with it. > But the application can use Jute tool to generate the classes on demand. > Or better yet, the SequenceFile record reader may be able to do that > automatically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-483) Enhance 'bin/hadoop job -submit <>'
[ https://issues.apache.org/jira/browse/MAPREDUCE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-483: - Assignee: (was: Arun C Murthy) It's fine to have oozie support this, but I don't see why MR shouldn't... > Enhance 'bin/hadoop job -submit <>' > --- > > Key: MAPREDUCE-483 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-483 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Priority: Minor > Attachments: HADOOP-808_1_20061208.patch, myjob.tgz > > > Today 'bin/hadoop job -submit' takes a config file and just does a > JobClient.submitJob(jobConf), this isn't flexible enough for cases where the > user wants to submit a job which calls his main(), does some work and then > calls 'JobClient.runJob', I propose we enhance this subcommand to handle that > scenario. This could be really useful for the time we decide to setup a > simple webpage with 'form' where the user uploads a job.xml & job.jar and we > can fire the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-444) Job should be able to specify whether task vm is 32 or 64 bit
[ https://issues.apache.org/jira/browse/MAPREDUCE-444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-444: - > Job should be able to specify whether task vm is 32 or 64 bit > - > > Key: MAPREDUCE-444 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-444 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Nigel Daley >Priority: Minor > > Perhaps a job should be able to specify whether it wants it's task VM's to be > 32 or 64 bit. This could be accomplished by the -d32 and -d64 java options > when the task VM is exec'd. This becomes important for native libs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-444) Job should be able to specify whether task vm is 32 or 64 bit
[ https://issues.apache.org/jira/browse/MAPREDUCE-444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-444. - Resolution: Fixed Fixed via MR-279. > Job should be able to specify whether task vm is 32 or 64 bit > - > > Key: MAPREDUCE-444 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-444 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Nigel Daley >Priority: Minor > > Perhaps a job should be able to specify whether it wants it's task VM's to be > 32 or 64 bit. This could be accomplished by the -d32 and -d64 java options > when the task VM is exec'd. This becomes important for native libs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187414#comment-13187414 ] Hadoop QA commented on MAPREDUCE-3669: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510793/MAPREDUCE-3669.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1616//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1616//console This message is automatically generated. > Getting a lot of PriviledgedActionException / SaslException when running a job > -- > > Key: MAPREDUCE-3669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3669.patch, MAPREDUCE-3669.patch > > > On a secure cluster, when running a job we are seeing a lot of > PriviledgedActionException / SaslExceptions. The job runs fine, its just the > jobclient can't connect to the AM to get the progress information. > Its in a very tight loop retrying while getting the exceptions. > snip of the client log is: > 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 > 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47 > 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 > d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02, > Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020 > 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 > for user1 on 10.10.10.10:8020 > 12/01/13 15:33:45 INFO security.TokenCache: Got dt for > hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020 > 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser > is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2 > 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application > application_1326410042859_0008 to > ResourceManager at rmhost.domain/10.10.10.11:8040 > 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008 > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: Fail > ed to find any > Kerberos tgt)] > 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to > the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided ( > Mechanism level: > Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187407#comment-13187407 ] Mahadev konar commented on MAPREDUCE-2450: -- +1 the patch looks good to me. > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.apache.hadoop.ipc.Server.channelWrite(Server.java:1195) > at org.apache.hadoop.ipc.Server.access$1900(Server.java:77) > at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:613) > at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:677) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:981) > > Note that the task actually seemed to commit - it didn't get speculatively > executed or anything. However, the job wasn't able to continue until this one > task was done. Both parties seem to think the channel was closed. How does > the channel get closed externa
[jira] [Updated] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2450: - Attachment: MAPREDUCE-2450.patch This is the patch we committed to hadoop-0.20.2xx, forward-ported to 0.23. No tests since it's hard to unit test this. > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.apache.hadoop.ipc.Server.channelWrite(Server.java:1195) > at org.apache.hadoop.ipc.Server.access$1900(Server.java:77) > at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:613) > at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:677) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:981) > > Note that the task actually seemed to commit - it didn't get speculatively > executed or anything. However, the job wasn't able to continue until this one > task was done. Both parties seem to think the channel was closed. How does >
[jira] [Updated] (MAPREDUCE-2450) Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2450: - Fix Version/s: (was: 0.24.0) Assignee: Rajesh Balamohan Target Version/s: 0.23.1 Status: Patch Available (was: Open) Thanks to Rajesh for the original patch. This helps 0.23 a *lot*, ran a number of benchmarks. > Calls from running tasks to TaskTracker methods sometimes fail and incur a > 60s timeout > -- > > Key: MAPREDUCE-2450 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2450 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Matei Zaharia >Assignee: Rajesh Balamohan > Attachments: HADOOP-5380.Y.20.branch.patch, HADOOP-5380.patch, > HADOOP_5380-Y.0.20.20x.patch, MAPREDUCE-2450.patch, mapreduce-2450.patch > > > I'm seeing some map tasks in my jobs take 1 minute to commit after they > finish the map computation. On the map side, the output looks like this: > > 2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot > initialize JVM Metrics with processName=MAP, sessionId= - already initialized > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 800 > 2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = > 300 > 2009-03-02 21:30:55,493 INFO org.apache.hadoop.mapred.MapTask: data buffer = > 239075328/298844160 > 2009-03-02 21:30:55,494 INFO org.apache.hadoop.mapred.MapTask: record buffer > = 786432/983040 > 2009-03-02 21:31:00,381 INFO org.apache.hadoop.mapred.MapTask: Starting flush > of map output > 2009-03-02 21:31:07,892 INFO org.apache.hadoop.mapred.MapTask: Finished spill > 0 > 2009-03-02 21:31:07,951 INFO org.apache.hadoop.mapred.TaskRunner: > Task:attempt_200903022127_0001_m_003163_0 is done. And is in the process of > commiting > 2009-03-02 21:32:07,949 INFO org.apache.hadoop.mapred.TaskRunner: > Communication exception: java.io.IOException: Call to /127.0.0.1:50311 failed > on local exception: java.nio.channels.ClosedChannelException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:765) > at org.apache.hadoop.ipc.Client.call(Client.java:733) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.ping(Unknown Source) > at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:525) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.nio.channels.ClosedChannelException > at > java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:167) > at > java.nio.channels.SelectableChannel.register(SelectableChannel.java:254) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:331) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.FilterInputStream.read(FilterInputStream.java:116) > at > org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:276) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > at java.io.DataInputStream.readInt(DataInputStream.java:370) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > 2009-03-02 21:32:07,953 INFO org.apache.hadoop.mapred.TaskRunner: Task > 'attempt_200903022127_0001_m_003163_0' done. > > In the TaskTracker log, it looks like this: > > 2009-03-02 21:31:08,110 WARN org.apache.hadoop.ipc.Server: IPC Server > Responder, call ping(attempt_200903022127_0001_m_003163_0) from > 127.0.0.1:56884: output error > 2009-03-02 21:31:08,111 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 10 on 50311 caught: java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)at > org.apache.hadoop.ipc.Server.channelWrite(Server.java:1195) > at org.apache.hadoop.ipc.Server.access$1900(Server.java:77) > at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:613) > at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:677) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:981) > > Note that the task actually seemed to commit - it didn't get speculatively > executed or anything.
[jira] [Updated] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3669: - Attachment: MAPREDUCE-3669.patch This patch removes all duplication. This is a little tricky, since we are extending BlockingInterface. > Getting a lot of PriviledgedActionException / SaslException when running a job > -- > > Key: MAPREDUCE-3669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3669.patch, MAPREDUCE-3669.patch > > > On a secure cluster, when running a job we are seeing a lot of > PriviledgedActionException / SaslExceptions. The job runs fine, its just the > jobclient can't connect to the AM to get the progress information. > Its in a very tight loop retrying while getting the exceptions. > snip of the client log is: > 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 > 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47 > 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 > d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02, > Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020 > 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 > for user1 on 10.10.10.10:8020 > 12/01/13 15:33:45 INFO security.TokenCache: Got dt for > hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020 > 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser > is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2 > 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application > application_1326410042859_0008 to > ResourceManager at rmhost.domain/10.10.10.11:8040 > 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008 > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: Fail > ed to find any > Kerberos tgt)] > 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to > the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided ( > Mechanism level: > Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3669: - Status: Patch Available (was: Open) > Getting a lot of PriviledgedActionException / SaslException when running a job > -- > > Key: MAPREDUCE-3669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3669.patch, MAPREDUCE-3669.patch > > > On a secure cluster, when running a job we are seeing a lot of > PriviledgedActionException / SaslExceptions. The job runs fine, its just the > jobclient can't connect to the AM to get the progress information. > Its in a very tight loop retrying while getting the exceptions. > snip of the client log is: > 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 > 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47 > 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 > d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02, > Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020 > 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 > for user1 on 10.10.10.10:8020 > 12/01/13 15:33:45 INFO security.TokenCache: Got dt for > hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020 > 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser > is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2 > 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application > application_1326410042859_0008 to > ResourceManager at rmhost.domain/10.10.10.11:8040 > 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008 > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: Fail > ed to find any > Kerberos tgt)] > 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to > the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided ( > Mechanism level: > Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3669: - Status: Open (was: Patch Available) Trying out a less redundant patch. > Getting a lot of PriviledgedActionException / SaslException when running a job > -- > > Key: MAPREDUCE-3669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3669.patch > > > On a secure cluster, when running a job we are seeing a lot of > PriviledgedActionException / SaslExceptions. The job runs fine, its just the > jobclient can't connect to the AM to get the progress information. > Its in a very tight loop retrying while getting the exceptions. > snip of the client log is: > 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 > 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47 > 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 > d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02, > Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020 > 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 > for user1 on 10.10.10.10:8020 > 12/01/13 15:33:45 INFO security.TokenCache: Got dt for > hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020 > 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser > is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2 > 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application > application_1326410042859_0008 to > ResourceManager at rmhost.domain/10.10.10.11:8040 > 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008 > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: Fail > ed to find any > Kerberos tgt)] > 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to > the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided ( > Mechanism level: > Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3607) Port missing new API mapreduce lib classes to 1.x
[
https://issues.apache.org/jira/browse/MAPREDUCE-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated MAPREDUCE-3607:
-
Attachment: MAPREDUCE-3607.patch
Here's a new patch which adds FieldSelectionMapper/Reducer, NLineInputFormat,
SequenceFile input/output formats, JobControl, and partition classes, along
with tests for all of the classes.
The results of test-patch:
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 100 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] -1 findbugs. The patch appears to introduce 5 new Findbugs
(version 1.3.9) warnings.
{noformat}
Note the findbugs warnings are present in trunk too, since this is a backport.
Tests pass.
I would like this to be considered for inclusion in 1.1.0.
> Port missing new API mapreduce lib classes to 1.x
> -
>
> Key: MAPREDUCE-3607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3607
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 1.0.0
>Reporter: Tom White
>Assignee: Tom White
> Attachments: MAPREDUCE-3607.patch, MAPREDUCE-3607.patch
>
>
> There are a number of classes under mapreduce.lib that are not present in the
> 1.x series. Including these would help users and downstream projects using
> the new MapReduce API migrate to later versions of Hadoop in the future.
> A few examples of where this would help:
> * Sqoop uses mapreduce.lib.db.DBWritable and
> mapreduce.lib.input.CombineFileInputFormat (SQOOP-384).
> * Mahout uses mapreduce.lib.output.MultipleOutputs (MAHOUT-822).
> * HBase has a backport of mapreduce.lib.partition.InputSampler and
> TotalOrderPartitioner (in org.apache.hadoop.hbase.mapreduce.hadoopbackport) -
> it would be better if it used the ones in Hadoop.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187313#comment-13187313
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3669:
Mahadev, this looks good overall. But I think we can reuse more code by making
{{HSClientProtocolPBClientImpl}} simply extend {{MRClientProtocolPBClientImpl}}
and similarly {{HSClientProtocolPBServiceImpl}} extending
{{MRClientProtocolPBServiceImpl}}. Right?
> Getting a lot of PriviledgedActionException / SaslException when running a job
> --
>
> Key: MAPREDUCE-3669
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Mahadev konar
>Priority: Blocker
> Attachments: MAPREDUCE-3669.patch
>
>
> On a secure cluster, when running a job we are seeing a lot of
> PriviledgedActionException / SaslExceptions. The job runs fine, its just the
> jobclient can't connect to the AM to get the progress information.
> Its in a very tight loop retrying while getting the exceptions.
> snip of the client log is:
> 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68
> 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47
> 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34
> d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02,
> Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020
> 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8
> for user1 on 10.10.10.10:8020
> 12/01/13 15:33:45 INFO security.TokenCache: Got dt for
> hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020
> 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
> 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2
> 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1326410042859_0008 to
> ResourceManager at rmhost.domain/10.10.10.11:8040
> 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008
> 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
> rmhost.domain:8088/proxy/application_1326410042859_0008/
> 12/01/13 15:33:52 ERROR security.UserGroupInformation:
> PriviledgedActionException as:us...@dev.ygrid.yahoo.com
> (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level: Fail
> ed to find any
> Kerberos tgt)]
> 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to
> the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 12/01/13 15:33:52 ERROR security.UserGroupInformation:
> PriviledgedActionException as:us...@dev.ygrid.yahoo.com
> (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException:
> GSS initiate failed [Caused by GSSException: No valid credentials provided (
> Mechanism level:
> Failed to find any Kerberos tgt)]
> 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job:
> rmhost.domain:8088/proxy/application_1326410042859_0008/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187267#comment-13187267 ] Hudson commented on MAPREDUCE-3641: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1567 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1567/]) MAPREDUCE-3641. Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232182 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3605) Allow mr commands to be run via bin/hadoop
[
https://issues.apache.org/jira/browse/MAPREDUCE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187265#comment-13187265
]
Mahadev konar commented on MAPREDUCE-3605:
--
@Prashanth,
Looks like you already had the patch. I uploaded one on HADOOP-7971. This
should probably be a hadoop common jira. I also removed jobtracker/tasktracker
and others not supported options from the patch.
> Allow mr commands to be run via bin/hadoop
> --
>
> Key: MAPREDUCE-3605
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3605
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Ramya Sunil
>Assignee: Prashant Sharma
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3605.patch
>
>
> MR command line options are not supported in bin/hadoop.
> {noformat}
> bin/hadoop job
> Exception in thread "main" java.lang.NoClassDefFoundError: job
> Caused by: java.lang.ClassNotFoundException: job
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: job. Program will exit.
> {noformat}
> A deprecated message like "DEPRECATED: Use of this script to execute mapred
> command is deprecated. Instead use the mapred command for it." should be
> displayed along with the correct output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187257#comment-13187257 ] Hudson commented on MAPREDUCE-3641: --- Integrated in Hadoop-Mapreduce-0.23-Commit #396 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/396/]) MAPREDUCE-3641. Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. Contributed by Arun C Murthy. svn merge --ignore-ancestry -c 1232182 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232183 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3657) State machine visualize build fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187252#comment-13187252
]
Hudson commented on MAPREDUCE-3657:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1566 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1566/])
MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe via
mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232167
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
> State machine visualize build fails
> ---
>
> Key: MAPREDUCE-3657
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3657
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3657.patch
>
>
> Attempting to build the state machine graphs with {{mvn -Pvisualize compile}}
> fails for the resourcemanager and nodemanager projects. The build fails
> because org.apache.commons.logging.LogFactory isn't in the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3562) Concurrency issues in MultipleOutputs,JobControl,Counters
[
https://issues.apache.org/jira/browse/MAPREDUCE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187250#comment-13187250
]
Hadoop QA commented on MAPREDUCE-3562:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507536/MAPREDUCE-3562.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1615//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1615//console
This message is automatically generated.
> Concurrency issues in MultipleOutputs,JobControl,Counters
> -
>
> Key: MAPREDUCE-3562
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3562
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Teja Ch N V
>Assignee: Ravi Teja Ch N V
> Attachments: MAPREDUCE-3562.patch
>
>
> bq.MultipleOutputs
> The close of recordwriters should be synchronized.
> public void close() throws IOException, InterruptedException {
> for (RecordWriter writer : recordWriters.values()) {
> writer.close(context);
> bq.JobControl.java
> the getters of the jobs to be synchronized.
> bq.Counters.java
>makeEscapedCompactString to be made synchronized.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey resolved MAPREDUCE-3664. - Resolution: Fixed Fix Version/s: 0.23.0 0.24.0 Committed to 23 as well. > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Assignee: Brandon Li >Priority: Minor > Fix For: 0.24.0, 0.23.0 > > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187243#comment-13187243 ] Hudson commented on MAPREDUCE-3641: --- Integrated in Hadoop-Hdfs-0.23-Commit #373 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/373/]) MAPREDUCE-3641. Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. Contributed by Arun C Murthy. svn merge --ignore-ancestry -c 1232182 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232183 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187237#comment-13187237 ] Hudson commented on MAPREDUCE-3641: --- Integrated in Hadoop-Common-0.23-Commit #383 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/383/]) MAPREDUCE-3641. Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. Contributed by Arun C Murthy. svn merge --ignore-ancestry -c 1232182 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232183 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187238#comment-13187238 ] Hudson commented on MAPREDUCE-3641: --- Integrated in Hadoop-Common-trunk-Commit #1549 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1549/]) MAPREDUCE-3641. Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232182 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187236#comment-13187236 ] Hudson commented on MAPREDUCE-3641: --- Integrated in Hadoop-Hdfs-trunk-Commit #1622 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1622/]) MAPREDUCE-3641. Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. Contributed by Arun C Murthy. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232182 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3641: --- Resolution: Fixed Fix Version/s: 0.23.1 Release Note: Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk and branch-023. Thanks Arun! > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3657) State machine visualize build fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187225#comment-13187225
]
Hudson commented on MAPREDUCE-3657:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #395 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/395/])
MAPREDUCE-3657. State machine visualize build fails. (Jason
LoweMAPREDUCE-3657. State machine visualize build fails. (Jason Lowe via
mahadev) - Merging r1232167 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232168
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
> State machine visualize build fails
> ---
>
> Key: MAPREDUCE-3657
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3657
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3657.patch
>
>
> Attempting to build the state machine graphs with {{mvn -Pvisualize compile}}
> fails for the resourcemanager and nodemanager projects. The build fails
> because org.apache.commons.logging.LogFactory isn't in the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187226#comment-13187226 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3641: +1. This looks good. I also have an offline confirmation from [~sseth] that this runs fine on a 350 node cluster and improves performance for a few benchmarks. May be good to fix FifoScheduler also to have this, but we can keep that separate. I am going to push this in. > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3363) The "totalnodes" and "memorytotal" fields show wrong information if the nodes are going down and coming up early(before 10min)
[ https://issues.apache.org/jira/browse/MAPREDUCE-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3363: - Affects Version/s: 0.23.0 > The "totalnodes" and "memorytotal" fields show wrong information if the > nodes are going down and coming up early(before 10min) > > > Key: MAPREDUCE-3363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Ramgopal N >Priority: Critical > Attachments: Applications.htm, screenshot-1.jpg > > > The node details is not moved from Totalnodes to lostnodes for 60 ms.So > if the node is going down and coming up before the expiry interval, the > cluster status in terms of the total nodes and Total cluster memory displays > wrong values. > Atleast, if the same node is coming up again...should not consider as new > node.No point of time duplicate nodes should be displayed in Totalnodes list. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3657) State machine visualize build fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187215#comment-13187215
]
Hudson commented on MAPREDUCE-3657:
---
Integrated in Hadoop-Hdfs-0.23-Commit #372 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/372/])
MAPREDUCE-3657. State machine visualize build fails. (Jason
LoweMAPREDUCE-3657. State machine visualize build fails. (Jason Lowe via
mahadev) - Merging r1232167 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232168
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
> State machine visualize build fails
> ---
>
> Key: MAPREDUCE-3657
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3657
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3657.patch
>
>
> Attempting to build the state machine graphs with {{mvn -Pvisualize compile}}
> fails for the resourcemanager and nodemanager projects. The build fails
> because org.apache.commons.logging.LogFactory isn't in the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3657) State machine visualize build fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187211#comment-13187211
]
Hudson commented on MAPREDUCE-3657:
---
Integrated in Hadoop-Common-trunk-Commit #1548 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1548/])
MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe via
mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232167
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
> State machine visualize build fails
> ---
>
> Key: MAPREDUCE-3657
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3657
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3657.patch
>
>
> Attempting to build the state machine graphs with {{mvn -Pvisualize compile}}
> fails for the resourcemanager and nodemanager projects. The build fails
> because org.apache.commons.logging.LogFactory isn't in the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3657) State machine visualize build fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187210#comment-13187210
]
Hudson commented on MAPREDUCE-3657:
---
Integrated in Hadoop-Common-0.23-Commit #382 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/382/])
MAPREDUCE-3657. State machine visualize build fails. (Jason
LoweMAPREDUCE-3657. State machine visualize build fails. (Jason Lowe via
mahadev) - Merging r1232167 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232168
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
> State machine visualize build fails
> ---
>
> Key: MAPREDUCE-3657
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3657
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3657.patch
>
>
> Attempting to build the state machine graphs with {{mvn -Pvisualize compile}}
> fails for the resourcemanager and nodemanager projects. The build fails
> because org.apache.commons.logging.LogFactory isn't in the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3657) State machine visualize build fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187209#comment-13187209
]
Hudson commented on MAPREDUCE-3657:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1621 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1621/])
MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe via
mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232167
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
> State machine visualize build fails
> ---
>
> Key: MAPREDUCE-3657
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3657
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3657.patch
>
>
> Attempting to build the state machine graphs with {{mvn -Pvisualize compile}}
> fails for the resourcemanager and nodemanager projects. The build fails
> because org.apache.commons.logging.LogFactory isn't in the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3679) AM logs and others should not automatically refresh after every 1 second.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3679: - Component/s: mrv2 > AM logs and others should not automatically refresh after every 1 second. > - > > Key: MAPREDUCE-3679 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3679 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar > > If you are looking through the logs for AM or containers, the page is > automatically refreshed after 1 second or so which makes it problematic to > search through the page or debug using the content on the page. We should not > refresh the logs page. There should be a button to manually refresh if the > user needs to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3657) State machine visualize build fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3657:
-
Resolution: Fixed
Fix Version/s: 0.23.1
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
The patch looks good. I just committed this. Thanks Jason!
> State machine visualize build fails
> ---
>
> Key: MAPREDUCE-3657
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3657
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Minor
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3657.patch
>
>
> Attempting to build the state machine graphs with {{mvn -Pvisualize compile}}
> fails for the resourcemanager and nodemanager projects. The build fails
> because org.apache.commons.logging.LogFactory isn't in the classpath.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187159#comment-13187159
]
Hudson commented on MAPREDUCE-3649:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1565 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1565/])
MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi
Prakash via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187151#comment-13187151
]
Hudson commented on MAPREDUCE-3649:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #394 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/394/])
MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi
Prakash via mahadev) - Merging r1232126 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187137#comment-13187137 ] Hudson commented on MAPREDUCE-3664: --- Integrated in Hadoop-Mapreduce-0.23-Commit #393 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/393/]) Merged r1230708 from trunk for MAPREDUCE-3664. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232124 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Assignee: Brandon Li >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187135#comment-13187135
]
Hudson commented on MAPREDUCE-3649:
---
Integrated in Hadoop-Common-0.23-Commit #381 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/381/])
MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi
Prakash via mahadev) - Merging r1232126 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187133#comment-13187133
]
Hudson commented on MAPREDUCE-3649:
---
Integrated in Hadoop-Common-trunk-Commit #1547 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1547/])
MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi
Prakash via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187131#comment-13187131
]
Hudson commented on MAPREDUCE-3649:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1620 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1620/])
MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi
Prakash via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232126
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187130#comment-13187130
]
Hudson commented on MAPREDUCE-3649:
---
Integrated in Hadoop-Hdfs-0.23-Commit #371 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/371/])
MAPREDUCE-3649. Job End notification gives an error on calling back. (Ravi
Prakash via mahadev) - Merging r1232126 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232128
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar resolved MAPREDUCE-3649.
--
Resolution: Fixed
Hadoop Flags: Reviewed
I just committed this. Thanks Ravi and Alejandro!
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-634) Jython WordCount.py example fails with Java >= 1.4
[
https://issues.apache.org/jira/browse/MAPREDUCE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187120#comment-13187120
]
Shannon -jj Behrens commented on MAPREDUCE-634:
---
> The rule is Java 1.6 and nothing lower. User-side issue, closing out.
That's funny. I filed this way back in '07 ;)
> Jython WordCount.py example fails with Java >= 1.4
> --
>
> Key: MAPREDUCE-634
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-634
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples
> Environment: Java versions >= 1.4.
>Reporter: Shannon -jj Behrens
>Priority: Minor
>
> I had a hard time getting the Jython WordCount.py example to work due to
> changes to Java:
>cd ~/Desktop/hadoop-0.9.2/src/examples/python
>bash compile
>1 ./jpywork/org/apache/hadoop/examples/getopt.java:268: as of release 1.4,
>'assert' is a keyword, and may not be used as an identifier
>(try -source 1.3 or lower to use 'assert' as an identifier)
>if (frame.getglobal("__debug__").__nonzero__())
>Py.assert(frame.getglobal("len").__call__(frame.getlocal(2))._eq(i$9));
> To work around the problem, I updated the compile file. I added
> '--compileropts "-source 1.3"' to the jythonc call. Perhaps the
> underlying problem is Jython-specific, but at least there's a
> workaround for us.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3675) A job must not be submitted when there are no mappers to run
[ https://issues.apache.org/jira/browse/MAPREDUCE-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-3675. -- Resolution: Invalid Target Version/s: 0.23.1, 0.24.0, 1.1.0, 0.22.1 (was: 0.22.1, 1.1.0, 0.24.0, 0.23.1) Agree with Vinod. Oozie workflows with such jobs is a particularly important use-case for this 'feature'. > A job must not be submitted when there are no mappers to run > > > Key: MAPREDUCE-3675 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3675 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 1.0.0 >Reporter: Harsh J >Priority: Minor > > Right now, one's able to run a job with an empty input directory and a job is > indeed scheduled. > The job runs no mappers, but any number of specified reducers are run anyway > as dummy tasks. > This should be/could be avoided. I do not see a use for such an allowance, > and it looks mostly like a logic slip to me with empty arrays involved and > accepted. > The fix could be simply in the job submission code, where we can avoid > submitting if the splits are nil. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187115#comment-13187115 ] Hudson commented on MAPREDUCE-3664: --- Integrated in Hadoop-Common-0.23-Commit #380 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/380/]) Merged r1230708 from trunk for MAPREDUCE-3664. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232124 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Assignee: Brandon Li >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187112#comment-13187112 ] Hudson commented on MAPREDUCE-3664: --- Integrated in Hadoop-Hdfs-0.23-Commit #370 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/370/]) Merged r1230708 from trunk for MAPREDUCE-3664. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1232124 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Assignee: Brandon Li >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3675) A job must not be submitted when there are no mappers to run
[ https://issues.apache.org/jira/browse/MAPREDUCE-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187110#comment-13187110 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3675: No maps jobs are valid use-cases. For jobs with no maps, and sometimes even no reducers, I know of customers who depend on the side-effects caused by setup/cleanup tasks and/or job-success state. This is generally useful in workflow scenarios which run periodically and when they run into cases where some job in the workflow doesn't have input data, but has to cause either some side-effect or just finish successfully so that downstream jobs can proceed without breakage. Given that, I think this should be closed as invalid. Because of the same, we are supporting 0maps/0reduces cases in mrv2 also. > A job must not be submitted when there are no mappers to run > > > Key: MAPREDUCE-3675 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3675 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 1.0.0 >Reporter: Harsh J >Priority: Minor > > Right now, one's able to run a job with an empty input directory and a job is > indeed scheduled. > The job runs no mappers, but any number of specified reducers are run anyway > as dummy tasks. > This should be/could be avoided. I do not see a use for such an allowance, > and it looks mostly like a logic slip to me with empty arrays involved and > accepted. > The fix could be simply in the job submission code, where we can avoid > submitting if the splits are nil. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3679) AM logs and others should not automatically refresh after every 1 second.
AM logs and others should not automatically refresh after every 1 second. - Key: MAPREDUCE-3679 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3679 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Mahadev konar If you are looking through the logs for AM or containers, the page is automatically refreshed after 1 second or so which makes it problematic to search through the page or debug using the content on the page. We should not refresh the logs page. There should be a button to manually refresh if the user needs to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187099#comment-13187099
]
Mahadev konar commented on MAPREDUCE-3649:
--
Nevermind,
I ran the patch with an oozie cluster. It works fine. Ill go ahead and check
this in.
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187096#comment-13187096 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3641: Looking through the patch. > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187087#comment-13187087
]
Mahadev konar commented on MAPREDUCE-3649:
--
@Ravi,
Did you test out the patch on a cluster?
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar reassigned MAPREDUCE-3649:
Assignee: Ravi Prakash (was: Jason Lowe)
Correcting the assignment.
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Ravi Prakash
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187083#comment-13187083 ] Brandon Li commented on MAPREDUCE-3664: --- Thanks for the reminder and approval. It will be checked into 0.23 soon. > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Assignee: Brandon Li >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3649) Job End notification gives an error on calling back.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar reassigned MAPREDUCE-3649:
Assignee: Jason Lowe (was: Mahadev konar)
> Job End notification gives an error on calling back.
>
>
> Key: MAPREDUCE-3649
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3649
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Mahadev konar
>Assignee: Jason Lowe
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3649.patch, MAPREDUCE-3649.patch
>
>
> When calling job end notification for oozie the AM fails with the following
> trace:
> {noformat}
> 2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log:
> Job end notification to
> http://HOST:11000/oozie/v0/callback?id=000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED&;
> failed
> java.net.UnknownServiceException: no content-type
> at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
> at java.net.URLConnection.getContent(URLConnection.java:689)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
> at
> org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:388)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:375)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3647) Pipes job fails with "Illegal text protocol"
[ https://issues.apache.org/jira/browse/MAPREDUCE-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar resolved MAPREDUCE-3647. -- Resolution: Not A Problem Looks like the issue has been fixed. No longer an issue. Ramya, please reopen if you see this again. > Pipes job fails with "Illegal text protocol" > > > Key: MAPREDUCE-3647 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3647 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, pipes >Affects Versions: 0.23.1 >Reporter: Ramya Sunil >Assignee: Mahadev konar >Priority: Blocker > Fix For: 0.23.1 > > > Pipes job fail with "Hadoop Pipes Exception: Illegal text protocol command" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3647) Pipes job fails with "Illegal text protocol"
[ https://issues.apache.org/jira/browse/MAPREDUCE-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3647: - Priority: Blocker (was: Critical) Assignee: Mahadev konar > Pipes job fails with "Illegal text protocol" > > > Key: MAPREDUCE-3647 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3647 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, pipes >Affects Versions: 0.23.1 >Reporter: Ramya Sunil >Assignee: Mahadev konar >Priority: Blocker > Fix For: 0.23.1 > > > Pipes job fail with "Hadoop Pipes Exception: Illegal text protocol command" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3678) The Map tasks logs should have the value of input split it processed
[ https://issues.apache.org/jira/browse/MAPREDUCE-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187044#comment-13187044 ] Harsh J commented on MAPREDUCE-3678: Task's own logs are the best place for this, not the daemons. The reason it is tedious to do/maintain at the framework level is that not all InputSplits may be FileSplits, and formats that do use FileSplits may use them in different ways as well (CombineFileIF, for instance). The InputSplit interface by itself is path-agnostic. > The Map tasks logs should have the value of input split it processed > > > Key: MAPREDUCE-3678 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3678 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: nodemanager, tasktracker >Affects Versions: 0.20.203.0, 0.20.205.0, 1.0.0 > Environment: Linux red hat. >Reporter: Bejoy KS > > It would be easier to debug some corner in tasks if we knew what was the > input split processed by that task. Map reduce task tracker log should > accommodate the same. Also in the jobdetails web UI, the split also should be > displayed along with the Split Locations. > Sample as > Input Split > hdfs://myserver:9000/userdata/sampleapp/inputdir/file1.csv - no>/ > This would be much beneficial to nail down some data quality issues in large > data volume processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187042#comment-13187042
]
Todd Lipcon commented on MAPREDUCE-3674:
hm, rather than replacing null with "", maybe better to just disallow it? That
way we're more likely to notice if we have a bug later.
{{Preconditions.checkNotNull}} or {{Preconditions.checkArgument}} perhaps.
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Attachments: MAPREDUCE-3674.patch
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>
> at org.apac
[jira] [Created] (MAPREDUCE-3678) The Map tasks logs should have the value of input split it processed
The Map tasks logs should have the value of input split it processed Key: MAPREDUCE-3678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3678 Project: Hadoop Map/Reduce Issue Type: New Feature Components: nodemanager, tasktracker Affects Versions: 1.0.0, 0.20.205.0, 0.20.203.0 Environment: Linux red hat. Reporter: Bejoy KS It would be easier to debug some corner in tasks if we knew what was the input split processed by that task. Map reduce task tracker log should accommodate the same. Also in the jobdetails web UI, the split also should be displayed along with the Split Locations. Sample as Input Split hdfs://myserver:9000/userdata/sampleapp/inputdir/file1.csv - / This would be much beneficial to nail down some data quality issues in large data volume processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187013#comment-13187013 ] Eli Collins commented on MAPREDUCE-3664: +1, it's wrong there as well > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Assignee: Brandon Li >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-218) Map/Reduce job with SequenceFileOutputFormat should be able to add user specified metadata to the output file
[ https://issues.apache.org/jira/browse/MAPREDUCE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-218. --- Resolution: Won't Fix Users may override/extend the SFOF if they want to utilize SequenceFile.Writer's new metadata methods/features. > Map/Reduce job with SequenceFileOutputFormat should be able to add user > specified metadata to the output file > - > > Key: MAPREDUCE-218 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-218 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Runping Qi > > When creating a map/reduce job with SequenceFileFormat, > the user would like to add some metada to the output files automatically. > In particular, if the output value class was a JuteRecord class generated > from a Jute IDL, > we would like to add JUTE_IDL/IDL_STRING as a attribute/value pair of the > metadata. > This way, the output files will be self describing: > When an application that tries to use the files may not have the value class > with it. > But the application can use Jute tool to generate the classes on demand. > Or better yet, the SequenceFile record reader may be able to do that > automatically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-575) Job completes but command doesn't return
[ https://issues.apache.org/jira/browse/MAPREDUCE-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-575. --- Resolution: Duplicate Dupe of HADOOP-4620 > Job completes but command doesn't return > > > Key: MAPREDUCE-575 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-575 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming > Environment: Linux, Java 1.6.0-b105 >Reporter: Andrew McNabb > > I've had a job submission command hang on many different occasions. I can't > tell exactly what makes it complete some times and hang others. Here's some > information about one time when it hanged. > I started a job at 12:40. Here is the info from 'ps aux' including the full > command line: > /ibrix/home/awm27/jdk1.6.0/bin/java -Xmx1000m > -Dhadoop.log.dir=/ibrix/home/awm27/hadoop/logs/log.f17880 > -Dhadoop.log.file=hadoop.log > -Dhadoop.home.dir=/ibrix/home/awm27/hadoop/bin/.. -Dhadoop.id.str= > -Dhadoop.root.logger=INFO,console > -Djava.library.path=/ibrix/home/awm27/hadoop/bin/../lib/native/Linux-amd64-64 > -classpath > /ibrix/home/awm27/hadoop/dynamic_conf/tmp.FkrUc17883:/ibrix/home/awm27/jdk1.6.0/lib/tools.jar:/ibrix/home/awm27/hadoop/bin/../build/classes:/ibrix/home/awm27/hadoop/bin/../build:/ibrix/home/awm27/hadoop/bin/../build/test/classes:/ibrix/home/awm27/hadoop/bin/..:/ibrix/home/awm27/hadoop/bin/../hadoop-0.10.1-core.jar:/ibrix/home/awm27/hadoop/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/ibrix/home/awm27/hadoop/bin/../lib/commons-codec-1.3.jar:/ibrix/home/awm27/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/ibrix/home/awm27/hadoop/bin/../lib/commons-logging-1.0.4.jar:/ibrix/home/awm27/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/ibrix/home/awm27/hadoop/bin/../lib/jets3t.jar:/ibrix/home/awm27/hadoop/bin/../lib/jetty-5.1.4.jar:/ibrix/home/awm27/hadoop/bin/../lib/junit-3.8.1.jar:/ibrix/home/awm27/hadoop/bin/../lib/log4j-1.2.13.jar:/ibrix/home/awm27/hadoop/bin/../lib/lucene-core-1.9.1.jar:/ibrix/home/awm27/hadoop/bin/../lib/servlet-api.jar:/ibrix/home/awm27/hadoop/bin/../lib/jetty-ext/ant.jar:/ibrix/home/awm27/hadoop/bin/../lib/jetty-ext/commons-el.jar:/ibrix/home/awm27/hadoop/bin/../lib/jetty-ext/jasper-compiler.jar:/ibrix/home/awm27/hadoop/bin/../lib/jetty-ext/jasper-runtime.jar:/ibrix/home/awm27/hadoop/bin/../lib/jetty-ext/jsp-api.jar > org.apache.hadoop.util.RunJar > /ibrix/home/awm27/hadoop/build/hadoop-streaming.jar -jobconf > mapred.job.name="MRPSO_RBF_2" -input MRPSO_RBF/MRPSO_RBF_1 -output > MRPSO_RBF/MRPSO_RBF_2 -mapper /ibrix/home/awm27/svn/mrpso/mapper.py -combiner > /ibrix/home/awm27/svn/mrpso/reducer.py -reducer > /ibrix/home/awm27/svn/mrpso/reducer.py -cmdenv > PYTHONPATH=/ibrix/home/awm27/svn/python -cmdenv MRPSO_FUNCTION=RBF -cmdenv > MRPSO_DATAFILES=/tmp/tmpT9M4cq > At the time of submission, the jobtracker reported receiving the job and > began processing it. The first line in this part of the logs is: > 2007-01-29 12:40:44,072 INFO org.apache.hadoop.mapred.JobInProgress: Choosing > cached task tip_0002_m_02 > At 13:16, the job completed, with the following normal log messages: > 2007-01-29 13:16:36,115 INFO org.apache.hadoop.mapred.JobInProgress: Task > 'task_0002_r_01_0' has completed tip_0002_r_01 successfully. > 2007-01-29 13:16:36,117 INFO org.apache.hadoop.mapred.TaskInProgress: Task > 'task_0002_r_01_0' has completed. > 2007-01-29 13:16:36,566 INFO org.apache.hadoop.mapred.JobInProgress: Task > 'task_0002_r_02_0' has completed tip_0002_r_02 successfully. > 2007-01-29 13:16:36,566 INFO org.apache.hadoop.mapred.TaskInProgress: Task > 'task_0002_r_02_0' has completed. > 2007-01-29 13:16:36,879 INFO org.apache.hadoop.mapred.JobInProgress: Task > 'task_0002_r_03_0' has completed tip_0002_r_03 successfully. > 2007-01-29 13:16:36,879 INFO org.apache.hadoop.mapred.TaskInProgress: Task > 'task_0002_r_03_0' has completed. > 2007-01-29 13:16:41,808 INFO org.apache.hadoop.mapred.JobInProgress: Task > 'task_0002_r_00_0' has completed tip_0002_r_00 successfully. > 2007-01-29 13:16:41,930 INFO org.apache.hadoop.mapred.TaskInProgress: Task > 'task_0002_r_00_0' has completed. > 2007-01-29 13:16:41,940 INFO org.apache.hadoop.mapred.JobInProgress: Job > job_0002 has completed successfully. > 2007-01-29 13:16:41,942 INFO org.apache.hadoop.mapred.JobTracker: Removed > completed task 'task_0002_m_02_0' from 'tracker_m4b-3-2.local:50050' > ... > 2007-01-29 13:16:50,660 INFO org.apache.hadoop.mapred.JobTracker: Removed > completed task 'task_0002_m_16_0' from 'tracker_m4b-3-8.local:50050' > 2007-01-29 13:16:50,661 INFO org.apache.hadoop.mapred.JobTracker: Removed > completed task 'task_0002_m_44_0' from 'tracker_m4b-3-8.local:50050' > 2007-01-29 13:16
[jira] [Resolved] (MAPREDUCE-201) Map directly to HDFS or reduce()
[ https://issues.apache.org/jira/browse/MAPREDUCE-201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-201. --- Resolution: Not A Problem This should've been closed out before but was not. Closing out now. > Map directly to HDFS or reduce() > > > Key: MAPREDUCE-201 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-201 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Environment: all >Reporter: Doug Judd > > For situations where you know that the output of the Map phase is already > aggregated (e.g. the input is the output of another Map-reduce job and map() > preserves the aggregation), then there should be a way to tell the framework > that this is the case so that it can pipe the map() output directly to the > reduce() function, or HDFS in the case of IdentityReducer. This will > probably require forcing the number of map tasks to equal the number of > reduce tasks. This will save the disk I/O required to generate intermediate > files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-260) control-c of the submitting program should kill the job
[ https://issues.apache.org/jira/browse/MAPREDUCE-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-260. --- Resolution: Won't Fix This has settled down as a feature for users now. We should not change this behavior, and I do not see harm in letting it be as-is. > control-c of the submitting program should kill the job > --- > > Key: MAPREDUCE-260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-260 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Owen O'Malley > > Currently, if you kill the process that submitted the job, the job continues. > The default behavior should be to kill the job if the launching process dies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-490) Allow printing of TaskEvents for a Job from command line
[
https://issues.apache.org/jira/browse/MAPREDUCE-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-490.
---
Resolution: Not A Problem
Already available as "job -events".
> Allow printing of TaskEvents for a Job from command line
>
>
> Key: MAPREDUCE-490
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-490
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Devaraj Das
>Priority: Minor
>
> It might make sense to have these commands:
> bin/hadoop job -numEvents : this will print the number of events
> that the job generated so far
> bin/hadoop job: this will print the events
> in the range {fromEvent, fromEvent + maxEvents} for the given job id (only
> the contents of the event objects, not the actual logs). For example, it
> might print
> event# 100, Map/Reduce, Success, http://foo.com:port/, taskId
> bin/hadoop job : This will print the
> contents of the logs (like what the JobClient does today)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-429) HADOOP-801 doesn't add property to hadoop-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-429. --- Resolution: Invalid This was not closed earlier. Closing out now, has gone stale. > HADOOP-801 doesn't add property to hadoop-default.xml > - > > Key: MAPREDUCE-429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-429 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Nigel Daley >Priority: Minor > Attachments: HADOOP-919.patch > > > HADOOP-801 (in fixing HADOOP-805) adds a new configuration option -- > jobclient.output.filter. This property should have also been added to > hadoop-default.xml with a default value of NONE (the current default of > FAILURE is too verbose with INFO exceptions). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-444) Job should be able to specify whether task vm is 32 or 64 bit
[ https://issues.apache.org/jira/browse/MAPREDUCE-444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-444. --- Resolution: Not A Problem Could be done with child opts/etc. I think. Closing out since this has not seen any demand nor does it appear to be useful from my experience. > Job should be able to specify whether task vm is 32 or 64 bit > - > > Key: MAPREDUCE-444 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-444 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Nigel Daley >Priority: Minor > > Perhaps a job should be able to specify whether it wants it's task VM's to be > 32 or 64 bit. This could be accomplished by the -d32 and -d64 java options > when the task VM is exec'd. This becomes important for native libs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-278) Proposal for redesign/refactoring of the JobTracker and TaskTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-278.
---
Resolution: Not A Problem
Closing out as a result of MAPREDUCE-279
> Proposal for redesign/refactoring of the JobTracker and TaskTracker
> ---
>
> Key: MAPREDUCE-278
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-278
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Arun C Murthy
>Assignee: Sharad Agarwal
> Attachments: Job_Tracker_FSM.pdf, mapred_as_dfa.patch
>
>
> During discussions on HADOOP-815 wrt some hard-to-maintain code on the
> JobTracker we all agreed that the current state-of-affairs there is brittle
> and merits some rework.
> Case in point: there are back-calls from TaskInProgress to JobTracker and
> from JobInProgress to JobTracker which mean that synchronization is quite
> involved and brittle, leading to issues like HADOOP-600. Also one is forced
> to lock several data-structures individually before certain operations
> (taskTrackers, trackerExpiryQueue, jobs etc.)
> Hence I'd like to present some early thoughts (which have undergone a quick
> iteration) on how we could do slightly better by a bit of
> redesign/refactoring, also during discussions with Owen on the same we agreed
> that HADOOP-554 is an integral part along the same direction... and I also
> feel that a good candidate to be done along with this is HADOOP-398 (mapred
> package refactoring).
> Context:
> -
> a) The unit of communication between the JobTracker & TaskTracker is a 'task'.
> b) Due to (a) the JobTracker maintains a bunch of information related on the
> 'taskid' i.e. taskidToTipMap, taskidToTrackerMap etc. and hence we need to
> update the JobTracker's data-structures via back-calls from TaskInProgress &
> JobInProgress where the context is available (complete/failed task,
> already-completed task etc.)
> c) This implies that we have a fairly elaborate and hard to maintain locking
> structures and also some redundant information in the JobTracker; making it
> harder to maintain.
> Overall at both the JobTracker & TaskTracker the concept of a 'job' is
> overshadowed by the 'task'; which I propose we fix.
> Proposal:
> --
> Here is the main flow of control:
> JobTracker -> JobInProgress -> TaskInProgress -> task_attempt
> The main idea is to break the existing nexus between the JobTracker &
> TaskInProgress/taskid by (I've put code for illustrative purposes only, and
> ignored pieces irrelevant to this discussion):
> a) Making the 'job' the primary unit of communication between JobTracker &
> TaskTracker.
> b) TaskTrackerStatus now looks like this:
> class TaskTrackerStatus {
> List jobStatuses; // the status of the 'jobs' running on a
> TaskTracker
> String getTrackerName();
> }
> class JobStatus {
> List taskStatuses; // the status of the 'tasks' belonging to
> a job
> JobId getJobId();
> }
> c) The JobTracker maintains only a single map of jobid -> JobInProgress, and
> mapping from taskTracker -> List
> Map allJobs;
> Map> trackerToJobsMap;
> d) The JobTracker delegates a bunch of responsibilities to the JobInProgress
> to reflect the fact the primary 'concept' in map/reduce is the 'job', thus
> empowering the JobInProgress class:
> class JobInProgress {
> TaskInProgress[] mapTasks;
> TaskInProgress[] reduceTasks;
>
> Map> trackerToTasksMap; // tracker -> tasks
> running
> Map> trackerToMarkedTasksMap; // tracker ->
> completed (success/failed/killed) task-attempt,
>
>//but the tracker doesn't know it yet
> void updateStatus(JobStatus jobStatus);
> MapOutputLocation[] getMapOutputLocations(int[] mapTasksNeeded, int
> reduce);
> TaskAttempt getTaskToRun(String taskTracker);
> List getTaskToKill(String taskTracker);
> }
>
> d) On receipt of TaskTrackerStatus from a tracker, the processeing of
> heartbeat looks like this:
> for (JobStatus jobStatus : taskTrackerStatus.getJobStatuses()) {
>JobInProgress job = allJobs.get(jobId);
>synchronized (job) {
> job.updateStatus(jobStatus);
> return (HeartbeatResponse(repsonseId,
>job.getTaskAttemptToRun(trackerName),
>job.getTaskToKill(trackerName)
> ));
> }
> }
>
> The big change is that the JobTracker delegates a lot of responsibility to
> the JobInProgress, we get away from all the complicated synchronization
> constructs: simply lock the JobInProgress object at all places via
> allJobs/trackerToJobsMap and we are done. This also enhances thr
[jira] [Resolved] (MAPREDUCE-634) Jython WordCount.py example fails with Java >= 1.4
[
https://issues.apache.org/jira/browse/MAPREDUCE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved MAPREDUCE-634.
---
Resolution: Not A Problem
The rule is Java 1.6 and nothing lower. User-side issue, closing out.
> Jython WordCount.py example fails with Java >= 1.4
> --
>
> Key: MAPREDUCE-634
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-634
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: examples
> Environment: Java versions >= 1.4.
>Reporter: Shannon -jj Behrens
>Priority: Minor
>
> I had a hard time getting the Jython WordCount.py example to work due to
> changes to Java:
>cd ~/Desktop/hadoop-0.9.2/src/examples/python
>bash compile
>1 ./jpywork/org/apache/hadoop/examples/getopt.java:268: as of release 1.4,
>'assert' is a keyword, and may not be used as an identifier
>(try -source 1.3 or lower to use 'assert' as an identifier)
>if (frame.getglobal("__debug__").__nonzero__())
>Py.assert(frame.getglobal("len").__call__(frame.getlocal(2))._eq(i$9));
> To work around the problem, I updated the compile file. I added
> '--compileropts "-source 1.3"' to the jythonc call. Perhaps the
> underlying problem is Jython-specific, but at least there's a
> workaround for us.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-163) TaskTracker's Jetty throws SocketException followed by IllegalStateException
[ https://issues.apache.org/jira/browse/MAPREDUCE-163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-163. --- Resolution: Duplicate Dupe of MAPREDUCE-5 > TaskTracker's Jetty throws SocketException followed by IllegalStateException > > > Key: MAPREDUCE-163 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-163 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Nigel Daley > > When running the sort benchmark, these exceptions (271 pairs of them) were > noted in the log of the task tracker that was "lost". > 2007-01-05 19:02:28,663 WARN org.apache.hadoop.mapred.TaskTracker: > getMapOutput(task_0001_m_009177_0,2415) failed : > java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > at java.net.SocketOutputStream.write(SocketOutputStream.java:136) > at > org.mortbay.http.ChunkingOutputStream.bypassWrite(ChunkingOutputStream.java:151) > at > org.mortbay.http.BufferedOutputStream.write(BufferedOutputStream.java:139) > at org.mortbay.http.HttpOutputStream.write(HttpOutputStream.java:423) > at org.mortbay.jetty.servlet.ServletOut.write(ServletOut.java:54) > at > org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1526) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) > at > org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) > at > org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) > at org.mortbay.http.HttpServer.service(HttpServer.java:954) > at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) > at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) > at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) > at > org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) > at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) > at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) > 2007-01-05 19:02:28,664 WARN /: > /mapOutput?map=task_0001_m_009177_0&reduce=2415: > java.lang.IllegalStateException: Committed > at > org.mortbay.jetty.servlet.ServletHttpResponse.resetBuffer(ServletHttpResponse.java:212) > at > org.mortbay.jetty.servlet.ServletHttpResponse.sendError(ServletHttpResponse.java:375) > at > org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1551) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) > at > org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) > at > org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) > at org.mortbay.http.HttpServer.service(HttpServer.java:954) > at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) > at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) > at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) > at > org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) > at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) > at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-296) job statistics should be displayed in the web/ui
[ https://issues.apache.org/jira/browse/MAPREDUCE-296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reassigned MAPREDUCE-296: - Assignee: (was: Owen O'Malley) Unassigning from Owen (Owen, assign back if this is still on your list), so that others can pick it up without worry of conflict. > job statistics should be displayed in the web/ui > > > Key: MAPREDUCE-296 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-296 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley > Labels: newbie > > It would be really nice, if the job page in the web/ui showed the time that: > 1. first map started > 2. last map finished > 3. last reduce finished shuffle > 4. last reduce finished sort > 5. last reduce finished -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-296) job statistics should be displayed in the web/ui
[ https://issues.apache.org/jira/browse/MAPREDUCE-296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186846#comment-13186846 ] Harsh J commented on MAPREDUCE-296: --- This should be done over MR2's web ui if anyone's attempting it now. > job statistics should be displayed in the web/ui > > > Key: MAPREDUCE-296 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-296 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Labels: newbie > > It would be really nice, if the job page in the web/ui showed the time that: > 1. first map started > 2. last map finished > 3. last reduce finished shuffle > 4. last reduce finished sort > 5. last reduce finished -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-586) Streaming reducers throw OutOfMemory for not so large inputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-586. --- Resolution: Invalid Was a user config issue, not a framework issue. Closing out. > Streaming reducers throw OutOfMemory for not so large inputs > > > Key: MAPREDUCE-586 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-586 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Reporter: Sanjay Dahiya > Attachments: NetbeansProfie.png > > > I am seeing OutOfMemoryError for moderate size inputs (~70 text files, 20k > each ) causing job to fail in streaming. For very small inputs it still > succeeds. Looking into details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-141) reduce % complete incorrect in webui
[ https://issues.apache.org/jira/browse/MAPREDUCE-141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-141. --- Resolution: Duplicate Dupe of HADOOP-5210 > reduce % complete incorrect in webui > > > Key: MAPREDUCE-141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-141 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Nigel Daley > > I have a running job that has 1002 reduces. Currently the JobTracker WebUI > has the following: > Kind %Complete Num Tasks Pending Running Complete Killed > Failures > map > reduce 100.00% 10020 2 10000 10 > So 2 reduces are still running but the % complete is 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-483) Enhance 'bin/hadoop job -submit <>'
[ https://issues.apache.org/jira/browse/MAPREDUCE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-483. --- Resolution: Not A Problem IMHO, Oozie is the best way to do such workflow stuff. We should close this one out over Apache Oozie (OOZIE) and given that this has grown stale. > Enhance 'bin/hadoop job -submit <>' > --- > > Key: MAPREDUCE-483 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-483 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Attachments: HADOOP-808_1_20061208.patch, myjob.tgz > > > Today 'bin/hadoop job -submit' takes a config file and just does a > JobClient.submitJob(jobConf), this isn't flexible enough for cases where the > user wants to submit a job which calls his main(), does some work and then > calls 'JobClient.runJob', I propose we enhance this subcommand to handle that > scenario. This could be really useful for the time we decide to setup a > simple webpage with 'form' where the user uploads a job.xml & job.jar and we > can fire the job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-568) Streaming causes a lot of broken pipes which leads to job failure
[ https://issues.apache.org/jira/browse/MAPREDUCE-568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-568. --- Resolution: Invalid This mostly happens with faults in the user scripts. Closing out as its not really a bug. We do not see this in regular streaming jobs all the time as the description seems to go. > Streaming causes a lot of broken pipes which leads to job failure > - > > Key: MAPREDUCE-568 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-568 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Reporter: Mahadev konar > > java.io.IOException: Broken pipe > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:77) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:215) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1247) > at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:88) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:215) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1247) > A lot of these errors occur in streaming. THis leads to job failure. I still > am not sure what the reason might be of these errors, but these erros occur > too often. I will try and invetigate more to see what the reason might be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-280) TextInputFormat should allow different treatment on carriage return char '\r'
[ https://issues.apache.org/jira/browse/MAPREDUCE-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-280. --- Resolution: Not A Problem TIF now has configurable record-delimiter support, and hence this can be closed. > TextInputFormat should allow different treatment on carriage return char '\r' > - > > Key: MAPREDUCE-280 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-280 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Runping Qi >Assignee: Owen O'Malley > > The current implementation treat '\r' and '\n' both as line breakers. > However, in some cases, it is desiable to strictly use '\n' as the solely > line breaker and treat '\r' as a part of data in a line. > One way to do this is to make readline function as a member function so that > the user can create a subclass to overwrite the function with the desired > behavior. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-412) JobConf needs better javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-412. --- Resolution: Duplicate Dupe of HADOOP-2046 > JobConf needs better javadoc > > > Key: MAPREDUCE-412 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-412 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Nigel Daley >Priority: Minor > > The org.apache.hadoop.mapred.JobConf class needs better javadoc comments. > In general terms, these guidelines should be followed: > - http://java.sun.com/j2se/javadoc/writingdoccomments/#styleguide > - document the unchecked exceptions that the caller might reasonably > want to catch > - use font appropriately (detailed in link above) > - document default values and appropriate configuration file properties > - comments, tags, links, etc used consistently across API > In particular, the javadoc for this class should at least answer these > questions: > - all constructors: what happens if the supplied class, file path, or > configuration doesn't exist or is null? > - JobConf() constructor: where are default values loaded from? > - JobConf(Configuration) constructor: indicate that this is effectively a > copy constructor > - JobConf(String) constructor: are there any format requirements on the > string? local and dfs paths allowed? > - setter methods that take a path: what happens if the path doesn't exist? > - getter methods returning objects: if value has not been set, is null > returned or something else? > - are JobConf objects immutable? If not, why not? > - setNum*Tasks: is this per TaskTracker? or something else? > - methods that take a Path: what if the path is relative? > - setWorkingDirectory: is this deleted at the end of a job? should it be > local or dfs? > - *KeepFailedTaskFiles: if they are kept, where can they be found? > - *KeepTaskFilesPattern: what does "the files" mean? if they are kept, > where can they be found? > - deleteLocalFiles: undoubtedly a risky operation. Need a good spec. Is > it a recursive delete? > - which configuration entries are mandatory (i.e. must be set before > submitting the job)? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-160) Final map task gets stuck
[ https://issues.apache.org/jira/browse/MAPREDUCE-160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-160. --- Resolution: Not A Problem Has not been seen recently. Issue has gone stale, so closing out. > Final map task gets stuck > - > > Key: MAPREDUCE-160 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-160 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: using latest trunk >Reporter: Johan Oskarsson >Assignee: Owen O'Malley >Priority: Critical > > I've seen numerous jobs lately where the final map task gets stuck, never > finishing. > The jobtracker doesn't reassign the task. A restart of the tasktracker solves > the issue and the job can finish. > In the web interface it turns up as: > task_0028_m_000534_0 node17.herd1 RUNNING 0.00%10-Nov-2006 12:21:12 > 10-Nov-2006 12:22:19 (1mins, 6sec) > Task failed to report status for 604 seconds. Killing. > Only exception I find in that tasktracker log is this (a few times): > java.nio.channels.ClosedChannelException > at > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294) > at > org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108) > at > org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-456) add a link to the dfs from job tracker WI
[ https://issues.apache.org/jira/browse/MAPREDUCE-456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-456. --- Resolution: Won't Fix Don't see why we ought to do this cause JT isn't related to HDFS in any way. Job XMLs still show up full FS URIs, so can be accessed from UI already. > add a link to the dfs from job tracker WI > - > > Key: MAPREDUCE-456 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-456 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Yoram Arnon >Assignee: Owen O'Malley >Priority: Minor > > add a link to the dfs from job tracker WI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-84) JobClient waitForCompletion() method sometimes throws an NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-84. -- Resolution: Not A Problem Has not been seen recently. Issue has mostly grown stale, closing out. > JobClient waitForCompletion() method sometimes throws an NPE > > > Key: MAPREDUCE-84 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-84 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Thomas Friol >Assignee: Owen O'Malley > > java.lang.NullPointerException > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.isComplete(JobClient.java:113) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.waitForCompletion(JobClient.java:128) > Does someone have an idea why this happens ? > Thanks for any help. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-127) Task trackers don't register with the job tracker until after they clean out their working directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-127. --- Resolution: Invalid Should we register before we clean then? Frankly I've not seen this slow startups recently, even with several of our users running huge loads of tasks it never has shown up given that we keep it clean along the way. Do reopen if this is still an issue. I do not think it is anymore, and has gone stale. > Task trackers don't register with the job tracker until after they clean out > their working directory > > > Key: MAPREDUCE-127 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-127 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > > When TaskTrackers are started, they immediately start deleting their working > directory, which can take 30+ minutes. Unfortunately, that means they don't > register themselves with the JobTracker for a long time, so it looks like the > cluster is "missing". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-100) Sporadic TestEmptyJobWithDFS failure due to NPE is JobTracker.submitJob()
[ https://issues.apache.org/jira/browse/MAPREDUCE-100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-100. --- Resolution: Not A Problem Gone stale, and test failure does not occur now. Closing out. > Sporadic TestEmptyJobWithDFS failure due to NPE is JobTracker.submitJob() > - > > Key: MAPREDUCE-100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-100 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Nigel Daley >Assignee: Owen O'Malley > Attachments: TEST-org.apache.hadoop.mapred.TestEmptyJobWithDFS.txt > > > org.apache.hadoop.mapred.TestEmptyJobWithDFS has failed a couple of times > (low reproducibility) with the following exception: > 2006-10-17 21:48:24,875 INFO ipc.Server (Server.java:run(516)) - Server > handler 2 on 50050 call error: java.io.IOException: > java.lang.NullPointerException > java.io.IOException: java.lang.NullPointerException > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1020) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:385) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:514) > Complete test log attached. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3677) If "hadoop.security.authorization" is set to true, NM is not starting.
If "hadoop.security.authorization" is set to true, NM is not starting. -- Key: MAPREDUCE-3677 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3677 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.0 Reporter: Ramgopal N I have the hadoop cluster setup with root user.Accidentally i have set hadoop.security.authorization to true.I have not set any permissions in policy.xml.When i am trying to start the NM with root user ...it is throwing the following error Exception in thread "main" java.lang.NoClassDefFoundError: nodemanager Caused by: java.lang.ClassNotFoundException: nodemanager at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:303) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) Could not find the main class: nodemanager. Program will exit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
