[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787330#action_12787330
]
V.V.Chaitanya Krishna commented on MAPREDUCE-1082:
--
Had an offline discussion with Hemanth and Vinod.
The issues that are being found here can all be solved with the fix to
MAPREDUCE-1110. So, resolving this issue as duplicate of it.
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt, MAPREDUCE-1082-2.patch,
> MAPREDUCE-1082-3.patch, MAPREDUCE-1082-4.patch
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12785899#action_12785899
]
Hemanth Yamijala commented on MAPREDUCE-1082:
-
Looking close. Some final comments:
- We are assuming the job statuses cannot be null in QueueInfo. I think we
should check this in setJobStatuses. If it is null, we can set an empty array.
- The test case should call APIs like setRootQueues. getQueue is not passing
through the code path change you made in JobTracker.getQueueInfoArray
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt, MAPREDUCE-1082-2.patch,
> MAPREDUCE-1082-3.patch
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783579#action_12783579
]
Hemanth Yamijala commented on MAPREDUCE-1082:
-
Have a few comments on this patch:
- I don't know why readFields in JobQueueInfo needs to be overridden. In all
the API, it is QueueInfo objects that are transferred over IPC, and I think it
should remain that way.
- I think the test case needs to be an end-to-end test by definition, as the
fix is both in JobQueueInfo as well as in the JobTracker, where when we
translate JobQueueInfos to QueueInfos, we fix the translation by walking the
entire hierarchy. I would suggest a test that brings up a MiniMRCluster with
hierarchical queues, submits a job to one of the queues and calls
Cluster.getRootQueues and verifies the returned QueueInfo information. We might
need a package private JobTracker.setQueueManager to enable setting up
hierarchical queues with a miniMRCluster.
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt, MAPREDUCE-1082-2.patch
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780548#action_12780548
]
Hadoop QA commented on MAPREDUCE-1082:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12424575/MAPREDUCE-1082-2.patch
against trunk revision 882470.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/147/console
This message is automatically generated.
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt, MAPREDUCE-1082-2.patch
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776923#action_12776923
]
Vinod K V commented on MAPREDUCE-1082:
--
Or should I say MAPREDUCE-861 broke the back-compatibility of the serialization.
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt, MAPREDUCE-1082-2.patch
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776918#action_12776918
]
Vinod K V commented on MAPREDUCE-1082:
--
This patch is changing the serialization format of the deprecated
{{JobQueueInfo}}. Not sure if it is/isn't OK.
In any case, back-compatibility of {{JobQueueInfo}} itself is broken after the
incompatible changes done by MAPREDUCE-777; for e.g. some of the public methods
are no longer available and have protected scope now.
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt, MAPREDUCE-1082-2.patch
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776308#action_12776308
]
Vinod K V commented on MAPREDUCE-1082:
--
Thinking more about it, I am fine with just fixing the NPE and the problem with
{{JobQueueInfo.readFields()}} to solve this issue. We can delay the cleanup for
later via MAPREDUCE-1110.
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1082) Command line UI for queues' information is broken with hierarchical queues.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775354#action_12775354
]
Vinod K V commented on MAPREDUCE-1082:
--
I've looked at the attached patch. It has some problems. The most important one
is that for converting {{QueueInfo}} obects to {{JobQueueInfo}} objects, this
patch creates a new {{JobQueueInfo}} objects everytime. This causes other tests
like {{TestQueueMangerRefresh}} to fail. This has also proved to be problematic
in the past too while working with MAPREDUCE-893.
Going back and summarizing the main cause for this bug, the NPE occurs while
trying to serialize a hierarchy of {{JobQueueInfo}} objects. {{JobQueueInfo}}
extends {{QueueInfo}} and so serialization code is accordingly inherited. So,
the problem is as follows:
Hierarchy before serialization:
{code}
JobQueueInfo-one
|-- JobQueueInfo-child-one
`-- JobQueueInfo-child-two
{code}
Deserialized hierarchy:
{code}
JobQueueInfo-one
|-- QueueInfo-child-one
`-- QueueInfo-child-two
{code}
This poses problem while traversing the deserializing hierarchy.
Two solutions possible:
- From the deserialized hierarchy, construct a proper hierarchy of
{{JobQueueInfo}}s and use this from then on.
- Or solve the central probem which is about fixing the inconsistent handling
of relation {{JobQueueInfo}} with {{QueueInfo}} objects.
Given that the central problem has already resulted in some issues before too,
I propose that once and for all we fix the problems with {{JobQueueInfo}} <->
{{QueueInfo}} inter-conversion as part of this JIRA issue. Thoughts?
> Command line UI for queues' information is broken with hierarchical queues.
> ---
>
> Key: MAPREDUCE-1082
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1082
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobtracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: V.V.Chaitanya Krishna
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1082-1.txt
>
>
> When the command "./bin/mapred --config ~/tmp/conf/ queue -list" is run, it
> just hangs. I can see the following in the JT logs:
> {code}
> 2009-10-08 13:19:26,762 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 5 caught: java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:217)
> at org.apache.hadoop.mapreduce.QueueInfo.write(QueueInfo.java:223)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:159)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:126)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:70)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:1074)
> at org.apache.hadoop.ipc.Server.access$2400(Server.java:77)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:983)
> {code}
> Same is the case with "./bin/mapred --config ~/tmp/conf/ queue -info
> "
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
