[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852249#action_12852249 ] rahul k singh commented on MAPREDUCE-1376: -- # GridmixTestUtils is not a useful abstraction. Its functionality should remain in/be added to TestGridmixSubmission I feel , it is clean . I have added more code for creation of staging directories in it so I thought TestGridmixSubmission was getting really big with some reusable components . I think we might use those components again in future. Also It is was taking more time get it back to TestGridmixSubmission , I understand that we could do it later as well but given the urgency and as there is no harm in pulling some reusable code I am leaving it as it is. # Since UserResolver can remain an abstract class (no need to extend any groups mapping), parseUserList can remain there. parseUserList is only being used by RoundRobinUserResolve to iam leaving the interface as it is. Support for varied user submission in Gridmix - Key: MAPREDUCE-1376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Chris Douglas Assignee: Chris Douglas Attachments: 1376-2-yhadoop-security.patch, 1376-3-yhadoop20.100.patch, 1376-4-yhadoop20.100.patch, 1376-5-yhadoop20-100.patch, 1376-yhadoop-security.patch, M1376-0.patch, M1376-1.patch, M1376-2.patch, M1376-3.patch, M1376-4.patch Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[
https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12844723#action_12844723
]
Chris Douglas commented on MAPREDUCE-1376:
--
The attached patch, even with security disabled, did not work for me using
{{RoundRobinUserResolver}} (user from trace replaced with {{traceuser}}, user
from userlist replaced with {{targetuser}}):
{noformat}
10/03/12 20:20:52 WARN gridmix.JobSubmitter: Failed to submit GRIDMIX00106 as
targetuser via traceuser
org.apache.hadoop.ipc.RemoteException: User: traceuser is not allowed to
impersonate targetuser
at org.apache.hadoop.ipc.Client.call(Client.java:873)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:222)
at org.apache.hadoop.mapred.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:360)
at org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:443)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:437)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:422)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:477)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:766)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:475)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
at org.apache.hadoop.mapred.gridmix.GridmixJob.call(GridmixJob.java:230)
at
org.apache.hadoop.mapred.gridmix.JobSubmitter$SubmitTask.run(JobSubmitter.java:119)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
Other feedback:
* The following should be removed:
** This method is not used:
{noformat}
+ private void throwException(Throwable exception) throws Throwable{
+throw exception;
+ }
{noformat}
** GridmixJob contains a few lines with empty statements (see {{doAs}} blocks)
** {{GridmixTestUtils}} is not a useful abstraction. Its functionality should
remain in/be added to {{TestGridmixSubmission}}
* Consider {{Collections.emptyList()}} instead of creating and returning new,
empty collections (e.g. {{EchoUserResolver}})
* {{EchoUserResolver}} only needs to extend {{ShellBasedUnixGroupsMapping}} for
the unit test, right? The group mapping isn't important for the type. A
separate group mapping class in test would be appropriate (presumably one
already exists)
* {{RoundRobinUserResolver::parseUserList}} should be protected so subclasses
may override it
* Since {{UserResolver}} can remain an abstract class (no need to extend any
groups mapping), {{parseUserList}} can remain there.
* The default policy should be {{REPLAY}}, not {{STRESS}}
* {{JobSubmitter}} should not re-resolve the resolved UGI before calling
{{buildSplits}}.
* It is not sufficient to fix the failure above, but the job is not submitted
in a {{doAs}} block.
Support for varied user submission in Gridmix
-
Key: MAPREDUCE-1376
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/gridmix
Reporter: Chris Douglas
Assignee: Chris Douglas
Attachments: 1376-2-yhadoop-security.patch,
1376-yhadoop-security.patch, M1376-0.patch, M1376-1.patch, M1376-2.patch,
M1376-3.patch, M1376-4.patch
Gridmix currently submits all synthetic jobs as the client user. It should be
possible to map users in the trace to a set of users appropriate for the
target cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12804002#action_12804002 ] Hadoop QA commented on MAPREDUCE-1376: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431174/M1376-4.patch against trunk revision 902272. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/283/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/283/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/283/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/283/console This message is automatically generated. Support for varied user submission in Gridmix - Key: MAPREDUCE-1376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Chris Douglas Assignee: Chris Douglas Attachments: M1376-0.patch, M1376-1.patch, M1376-2.patch, M1376-3.patch, M1376-4.patch Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[
https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12804007#action_12804007
]
Chris Douglas commented on MAPREDUCE-1376:
--
The failing test, {{TestTaskFail.testWithDFS}} is not related.
Support for varied user submission in Gridmix
-
Key: MAPREDUCE-1376
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/gridmix
Reporter: Chris Douglas
Assignee: Chris Douglas
Attachments: M1376-0.patch, M1376-1.patch, M1376-2.patch,
M1376-3.patch, M1376-4.patch
Gridmix currently submits all synthetic jobs as the client user. It should be
possible to map users in the trace to a set of users appropriate for the
target cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12800658#action_12800658 ] Hadoop QA commented on MAPREDUCE-1376: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430363/M1376-1.patch against trunk revision 899501. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/389/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/389/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/389/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/389/console This message is automatically generated. Support for varied user submission in Gridmix - Key: MAPREDUCE-1376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Chris Douglas Assignee: Chris Douglas Attachments: M1376-0.patch, M1376-1.patch Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12800993#action_12800993 ] Hong Tang commented on MAPREDUCE-1376: -- Patch looks good mostly. A few comments: - Please open a jira to enhance rumen to extract UGI mapping information from job traces. - There is no API documentation of the interface methods in UserResolver. - There seems to be a typo in UserResolver javadoc Maps users to a set of users on the test cluster. = Maps users to a set of groups on the test cluster. - The interface of UserResolver.setTargetUsers() requires the userspec to be stored in a file. How about changing it to setTargetUsers(ListUserGroupInformation, Configuration conf)? Also currently EchoUserResolver/SubmitterUserResolver ignores this info, maybe we can have the method to return a boolean to indicate whether the method call takes effect or not? This would also require UserResolver.parseUserList moved to GridMix.run(). - Newly added conf parameters are not updated in GridMix.printUsage() method. - Any reason we split GridMix.run() to GridMix.run() and GridMix.start() now? Support for varied user submission in Gridmix - Key: MAPREDUCE-1376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Chris Douglas Assignee: Chris Douglas Attachments: M1376-0.patch, M1376-1.patch Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12801113#action_12801113 ] Hadoop QA commented on MAPREDUCE-1376: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430483/M1376-2.patch against trunk revision 899844. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/274/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/274/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/274/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/274/console This message is automatically generated. Support for varied user submission in Gridmix - Key: MAPREDUCE-1376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Chris Douglas Assignee: Chris Douglas Attachments: M1376-0.patch, M1376-1.patch, M1376-2.patch Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12800516#action_12800516 ] Hong Tang commented on MAPREDUCE-1376: -- Nice feature to have. This would be useful for evaluating fairness or potential starvation problems of schedulers like CapacityScheduler. Support for varied user submission in Gridmix - Key: MAPREDUCE-1376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Chris Douglas Assignee: Chris Douglas Attachments: M1376-0.patch Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1376) Support for varied user submission in Gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12800556#action_12800556 ] Hadoop QA commented on MAPREDUCE-1376: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12430343/M1376-0.patch against trunk revision 899501. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/271/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/271/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/271/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/271/console This message is automatically generated. Support for varied user submission in Gridmix - Key: MAPREDUCE-1376 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Reporter: Chris Douglas Assignee: Chris Douglas Attachments: M1376-0.patch Gridmix currently submits all synthetic jobs as the client user. It should be possible to map users in the trace to a set of users appropriate for the target cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
