[jira] Commented: (MAPREDUCE-806) WordCount example does not compile given the current instructions
[ https://issues.apache.org/jira/browse/MAPREDUCE-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988411#action_12988411 ] Muhammad Khojaye commented on MAPREDUCE-806: I am having the same issue. There is no jar exist with the above name. > WordCount example does not compile given the current instructions > - > > Key: MAPREDUCE-806 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-806 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples >Affects Versions: 0.20.1 > Environment: linux >Reporter: Hector Yuen >Priority: Trivial > > http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Example%3A+WordCount+v1.0 > In this example, the classpath is missing commons-cli-2.0-SNAPSHOT.jar > If we compile according to the instructions: > $ javac -classpath /hadoop/core/hadoop-0.20.0-core.jar -d ioperf_classes/ > src/WordCount.java > src/WordCount.java:54: cannot access org.apache.commons.cli.Options > class file for org.apache.commons.cli.Options not found > String[] otherArgs = new GenericOptionsParser(conf, > args).getRemainingArgs(); > The correct compilation should be > $ javac -classpath > /hadoop/core/hadoop-0.20.0-core.jar:/hadoop/core/lib/commons-cli-2.0-SNAPSHOT.jar > -d ioperf_classes/ src/WordCount.java -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2290) TestTaskCommit missing getProtocolSignature override
[ https://issues.apache.org/jira/browse/MAPREDUCE-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-2290: --- Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Thanks Nigel. I've committed this. > TestTaskCommit missing getProtocolSignature override > > > Key: MAPREDUCE-2290 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2290 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: mapred-2290-1.patch > > > Fixes an MR compilation error, HADOOP-6904 added a new implementation of > getProtocolSignature but TestTaskCommit doesn't override it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Closed: (MAPREDUCE-2290) TestTaskCommit missing getProtocolSignature override
[ https://issues.apache.org/jira/browse/MAPREDUCE-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins closed MAPREDUCE-2290. -- > TestTaskCommit missing getProtocolSignature override > > > Key: MAPREDUCE-2290 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2290 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.23.0 > > Attachments: mapred-2290-1.patch > > > Fixes an MR compilation error, HADOOP-6904 added a new implementation of > getProtocolSignature but TestTaskCommit doesn't override it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2290) TestTaskCommit missing getProtocolSignature override
[ https://issues.apache.org/jira/browse/MAPREDUCE-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988407#action_12988407 ] Nigel Daley commented on MAPREDUCE-2290: +1. please commit. > TestTaskCommit missing getProtocolSignature override > > > Key: MAPREDUCE-2290 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2290 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: mapred-2290-1.patch > > > Fixes an MR compilation error, HADOOP-6904 added a new implementation of > getProtocolSignature but TestTaskCommit doesn't override it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2290) TestTaskCommit missing getProtocolSignature override
[ https://issues.apache.org/jira/browse/MAPREDUCE-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-2290: --- Status: Patch Available (was: Open) > TestTaskCommit missing getProtocolSignature override > > > Key: MAPREDUCE-2290 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2290 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: mapred-2290-1.patch > > > Fixes an MR compilation error, HADOOP-6904 added a new implementation of > getProtocolSignature but TestTaskCommit doesn't override it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2290) TestTaskCommit missing getProtocolSignature override
[ https://issues.apache.org/jira/browse/MAPREDUCE-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-2290: --- Attachment: mapred-2290-1.patch Patch attached. Fixes the compilation error and TestTaskCommit passes. > TestTaskCommit missing getProtocolSignature override > > > Key: MAPREDUCE-2290 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2290 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: mapred-2290-1.patch > > > Fixes an MR compilation error, HADOOP-6904 added a new implementation of > getProtocolSignature but TestTaskCommit doesn't override it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2290) TestTaskCommit missing getProtocolSignature override
TestTaskCommit missing getProtocolSignature override Key: MAPREDUCE-2290 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2290 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Eli Collins Assignee: Eli Collins Fixes an MR compilation error, HADOOP-6904 added a new implementation of getProtocolSignature but TestTaskCommit doesn't override it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2285) MiniMRCluster does not start after ant test-patch
[
https://issues.apache.org/jira/browse/MAPREDUCE-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nigel Daley updated MAPREDUCE-2285:
---
Fix Version/s: 0.23.0
0.22.0
Status: Patch Available (was: Open)
Making patch available and will manually run thru hudson to test MR precommit.
> MiniMRCluster does not start after ant test-patch
> -
>
> Key: MAPREDUCE-2285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Ramkumar Vadali
>Priority: Blocker
> Fix For: 0.22.0, 0.23.0
>
> Attachments: cp-bad, cp-good, fix-build.diff
>
>
> Any test using MiniMRCluster hangs in the MiniMRCluster constructor after
> running ant test-patch. Steps to reproduce:
> 1. ant -Dpatch.file= -Dforrest.home= forrest> -Dfindbugs.home= -Dscratch.dir=/tmp/testpatch
> -Djava5.home= test-patch
> 2. Run any test that creates MiniMRCluster, say ant test
> -Dtestcase=TestFileArgs (contrib/streaming)
> Expected result: Test should succeed
> Actual result: Test hangs in MiniMRCluster.. This does not happen if
> we run ant clean after ant test-patch
> Test output:
> {code}
> [junit] 11/01/27 12:11:43 INFO ipc.Server: IPC Server handler 3 on 58675:
> starting
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: TaskTracker up at:
> localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: Starting tracker
> tracker_host0.foo.com:localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:44 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 0 time(s).
> [junit] 11/01/27 12:11:45 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 1 time(s).
> [junit] 11/01/27 12:11:46 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 2 time(s).
> [junit] 11/01/27 12:11:47 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 3 time(s).
> [junit] 11/01/27 12:11:48 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 4 time(s).
> [junit] 11/01/27 12:11:49 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 5 time(s).
> [junit] 11/01/27 12:11:50 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 6 time(s).
> [junit] 11/01/27 12:11:51 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 7 time(s).
> [junit] 11/01/27 12:11:52 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 8 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 9 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.RPC: Server at localhost/127.0.0.1:0
> not available yet, Z...
> {code}
> Stack trace:
> {code}
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:611)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:429)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:504)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
> at org.apache.hadoop.ipc.Client.call(Client.java:1008)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
> at org.apache.hadoop.mapred.$Proxy11.getProtocolVersion(Unknown
> Source)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:235)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:275)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:206)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:185)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:169)
> at org.apache.hadoop.mapred.TaskTracker$2.run(TaskTracker.java:699)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1142)
> at
> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:695)
> - locked <0x7f3b8ccc3870> (a org.apache.hadoop.mapred.TaskTracker)
> at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1391)
> at
[jira] Commented: (MAPREDUCE-2285) MiniMRCluster does not start after ant test-patch
[
https://issues.apache.org/jira/browse/MAPREDUCE-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988359#action_12988359
]
Giridharan Kesavan commented on MAPREDUCE-2285:
---
Patch looks good.
{quote}
After removing this we might be missing some dependencies which we'll have to
add to the specific projects ivy.xml files,
{quote}
I agree with Todd's point about adding deps to the ivy.xml.
> MiniMRCluster does not start after ant test-patch
> -
>
> Key: MAPREDUCE-2285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Ramkumar Vadali
>Priority: Blocker
> Attachments: cp-bad, cp-good, fix-build.diff
>
>
> Any test using MiniMRCluster hangs in the MiniMRCluster constructor after
> running ant test-patch. Steps to reproduce:
> 1. ant -Dpatch.file= -Dforrest.home= forrest> -Dfindbugs.home= -Dscratch.dir=/tmp/testpatch
> -Djava5.home= test-patch
> 2. Run any test that creates MiniMRCluster, say ant test
> -Dtestcase=TestFileArgs (contrib/streaming)
> Expected result: Test should succeed
> Actual result: Test hangs in MiniMRCluster.. This does not happen if
> we run ant clean after ant test-patch
> Test output:
> {code}
> [junit] 11/01/27 12:11:43 INFO ipc.Server: IPC Server handler 3 on 58675:
> starting
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: TaskTracker up at:
> localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: Starting tracker
> tracker_host0.foo.com:localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:44 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 0 time(s).
> [junit] 11/01/27 12:11:45 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 1 time(s).
> [junit] 11/01/27 12:11:46 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 2 time(s).
> [junit] 11/01/27 12:11:47 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 3 time(s).
> [junit] 11/01/27 12:11:48 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 4 time(s).
> [junit] 11/01/27 12:11:49 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 5 time(s).
> [junit] 11/01/27 12:11:50 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 6 time(s).
> [junit] 11/01/27 12:11:51 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 7 time(s).
> [junit] 11/01/27 12:11:52 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 8 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 9 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.RPC: Server at localhost/127.0.0.1:0
> not available yet, Z...
> {code}
> Stack trace:
> {code}
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:611)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:429)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:504)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
> at org.apache.hadoop.ipc.Client.call(Client.java:1008)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
> at org.apache.hadoop.mapred.$Proxy11.getProtocolVersion(Unknown
> Source)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:235)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:275)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:206)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:185)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:169)
> at org.apache.hadoop.mapred.TaskTracker$2.run(TaskTracker.java:699)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1142)
> at
> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:695)
> - locked <0x7f3b8ccc3870> (a org.apache.hadoop.mapred.TaskTracker)
> at org.a
[jira] Commented: (MAPREDUCE-2254) Allow setting of end-of-record delimiter for TextInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988349#action_12988349 ] Ahmed Radwan commented on MAPREDUCE-2254: - Thanks Todd for your comments. I have moved the changes to the LineReader, with new constructors for the user-specified line delimiter. If old constructors are used, then the exact old behavior is preserved. I have attached the new revised patches. > Allow setting of end-of-record delimiter for TextInputFormat > > > Key: MAPREDUCE-2254 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2254 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Radwan > Attachments: MAPREDUCE-2245.patch, MAPREDUCE-2254_r2.patch > > > It will be useful to allow setting the end-of-record delimiter for > TextInputFormat. The current implementation hardcodes '\n', '\r' or '\r\n' as > the only possible record delimiters. This is a problem if users have embedded > newlines in their data fields (which is pretty common). This is also a > problem for other tools using this TextInputFormat (See for example: > https://issues.apache.org/jira/browse/PIG-836 and > https://issues.cloudera.org/browse/SQOOP-136). > I have wrote a patch to address this issue. This patch allows users to > specify any custom end-of-record delimiter using a new added configuration > property. For backward compatibility, if this new configuration property is > absent, then the same exact previous delimiters are used (i.e., '\n', '\r' or > '\r\n'). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2254) Allow setting of end-of-record delimiter for TextInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-2254: Attachment: MAPREDUCE-2254_r2.patch > Allow setting of end-of-record delimiter for TextInputFormat > > > Key: MAPREDUCE-2254 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2254 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Radwan > Attachments: MAPREDUCE-2245.patch, MAPREDUCE-2254_r2.patch > > > It will be useful to allow setting the end-of-record delimiter for > TextInputFormat. The current implementation hardcodes '\n', '\r' or '\r\n' as > the only possible record delimiters. This is a problem if users have embedded > newlines in their data fields (which is pretty common). This is also a > problem for other tools using this TextInputFormat (See for example: > https://issues.apache.org/jira/browse/PIG-836 and > https://issues.cloudera.org/browse/SQOOP-136). > I have wrote a patch to address this issue. This patch allows users to > specify any custom end-of-record delimiter using a new added configuration > property. For backward compatibility, if this new configuration property is > absent, then the same exact previous delimiters are used (i.e., '\n', '\r' or > '\r\n'). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1825) jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps and finishedReduces when job is not initialized
[ https://issues.apache.org/jira/browse/MAPREDUCE-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988338#action_12988338 ] Scott Chen commented on MAPREDUCE-1825: --- > 2) FairSchedulerServlet is locking up in showPools method as soon as it tries > to synchronize on "scheduler". It continues again as soon as JIB.initTasks() > finish. Not sure how this is happening as initTasks() lock JIB's monitor and > showPools lock scheduler's monitor. Anyway, the patch doesn't seem to address > the FSS issue. Thanks for the testing, Priyo. Can you take a jstack at the moment that it waits for scheduler? Then we can figure out the lock dependency. > jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps > and finishedReduces when job is not initialized > --- > > Key: MAPREDUCE-1825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1825 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1825.txt, MAPREDUCE-1825_1.txt, > MAPREDUCE-1825_2.txt > > > JobInProgress.finishedMaps() and finishedReduces() are synchronized. They are > called from jobqueue_details.jsp and FairSchedulerServelet which iterates > through all jobs. If any job is in initialization, these pages don't come up > until the initialization finishes. > See > [comment|https://issues.apache.org/jira/browse/MAPREDUCE-1354?focusedCommentId=12834139&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12834139] > for more details -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2287) add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles
[ https://issues.apache.org/jira/browse/MAPREDUCE-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-2287: -- Status: Open (was: Patch Available) > add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT > web-consoles > -- > > Key: MAPREDUCE-2287 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2287 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: security >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ha-mapred.patch > > > This JIRA is for the MAPRED portion of HADOOP-7119 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-2263) MapReduce side of HADOOP-6904
[ https://issues.apache.org/jira/browse/MAPREDUCE-2263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang resolved MAPREDUCE-2263. -- Resolution: Fixed Hadoop Flags: [Reviewed] > MapReduce side of HADOOP-6904 > - > > Key: MAPREDUCE-2263 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2263 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: mapreduceRPC.patch > > > Make changes in Map/Reduce to incorporate HADOOP-6904. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2263) MapReduce side of HADOOP-6904
[ https://issues.apache.org/jira/browse/MAPREDUCE-2263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988302#action_12988302 ] Hairong Kuang commented on MAPREDUCE-2263: -- Since HADOOP-6904 breaks the compilation of Map/Reduce. This patch can not go through hudson. I just committed the patch without going through hudson. > MapReduce side of HADOOP-6904 > - > > Key: MAPREDUCE-2263 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2263 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: mapreduceRPC.patch > > > Make changes in Map/Reduce to incorporate HADOOP-6904. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2254) Allow setting of end-of-record delimiter for TextInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988299#action_12988299 ] Todd Lipcon commented on MAPREDUCE-2254: It seems you're being inconsistent here - why is it that LineReader shouldn't take an arbitrary delimiter but LineRecordReader should? What I mean here is that either the concept of a "line" is a sequence of characters with a newline, or it's a sequence of characters with an arbitrary delimiter. If "line" means something with a newline, then maybe this new feature should go in a new class like DelimitedTextInputFormat or something? If "line" really could be delimited by anything, then I would support moving this support up to LineReader, with a different constructor. That way at least the similar code will be next to each other. It just smells really bad to me to extend a class and then reimplement its only nontrivial method. Maybe we could alternatively extract an interface here? > Allow setting of end-of-record delimiter for TextInputFormat > > > Key: MAPREDUCE-2254 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2254 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Radwan > Attachments: MAPREDUCE-2245.patch > > > It will be useful to allow setting the end-of-record delimiter for > TextInputFormat. The current implementation hardcodes '\n', '\r' or '\r\n' as > the only possible record delimiters. This is a problem if users have embedded > newlines in their data fields (which is pretty common). This is also a > problem for other tools using this TextInputFormat (See for example: > https://issues.apache.org/jira/browse/PIG-836 and > https://issues.cloudera.org/browse/SQOOP-136). > I have wrote a patch to address this issue. This patch allows users to > specify any custom end-of-record delimiter using a new added configuration > property. For backward compatibility, if this new configuration property is > absent, then the same exact previous delimiters are used (i.e., '\n', '\r' or > '\r\n'). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2260) Remove auto-generated native build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988287#action_12988287 ] Roman Shaposhnik commented on MAPREDUCE-2260: - To finish up my previous answer -- the command used to build hadoop-mapreduce was: ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop-0.20/conf -Dlibrecordio=true veryclean api-report task-controller compile-c++ tar Finally, as for testing -- examples (pipes) were executed > Remove auto-generated native build files > > > Key: MAPREDUCE-2260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2260.diff > > Original Estimate: 24h > Remaining Estimate: 24h > > The repo currently includes the automake and autoconf generated files for the > native build. Per discussion on HADOOP-6421 let's remove them and use the > host's automake and autoconf. We should also do this for libhdfs and > fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2260) Remove auto-generated native build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988276#action_12988276 ] Roman Shaposhnik commented on MAPREDUCE-2260: - bq. it would be good to verify that ... ant tar produces a tarball which contains the generated files (ie a dist style tarball). Since one has to specify extra properties in order for native bits to be included in a dist style tarball here's a list of builds that were done: cd hadoop-common ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf veryclean api-report compile-core-native tar cd hadoop-hdfs ; ant -Dcompile.native=true -Dcompile.c++=true -Djava5.home=$JAVA5_HOME -Dforrest.home=$FORREST_HOME -Dhadoop.conf.dir=/etc/hadoop/conf -Dlibhdfs=1 veryclean api-report tar cd hadoop-mapreduce ; That resulted in the following native bits being created: c++/Linux-amd64-64/include/hadoop/Pipes.hh c++/Linux-amd64-64/include/hadoop/SerialUtils.hh c++/Linux-amd64-64/include/hadoop/TemplateFactory.hh c++/Linux-amd64-64/include/hadoop/StringUtils.hh c++/Linux-amd64-64/lib/libhadooppipes.a c++/Linux-amd64-64/lib/libhadooputils.a c++/Linux-amd64-64/lib/libhdfs.a c++/Linux-amd64-64/lib/libhdfs.so.0.0.0 c++/Linux-amd64-64/lib/libhdfs.so.0 c++/Linux-amd64-64/lib/libhdfs.so c++/Linux-amd64-64/lib/libhdfs.la c++/lib/libhdfs.a c++/lib/libhdfs.so.0.0.0 c++/lib/libhdfs.so.0 c++/lib/libhdfs.so c++/lib/libhdfs.la librecordio/librecordio.a bin/task-controller lib/native/Linux-amd64-64/libhadoop.so lib/native/Linux-amd64-64/libhadoop.a lib/native/Linux-amd64-64/libhadoop.so.1 lib/native/Linux-amd64-64/libhadoop.so.1.0.0 lib/native/Linux-amd64-64/libhadoop.la > Remove auto-generated native build files > > > Key: MAPREDUCE-2260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2260 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: MAPREDUCE-2260.diff > > Original Estimate: 24h > Remaining Estimate: 24h > > The repo currently includes the automake and autoconf generated files for the > native build. Per discussion on HADOOP-6421 let's remove them and use the > host's automake and autoconf. We should also do this for libhdfs and > fuse-dfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1915) IndexCache - getIndexInformation - check reduce index Out Of Bounds
[ https://issues.apache.org/jira/browse/MAPREDUCE-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyo Mustafi updated MAPREDUCE-1915: - Fix Version/s: 0.22.0 Status: Patch Available (was: Open) Added to 0.22 > IndexCache - getIndexInformation - check reduce index Out Of Bounds > --- > > Key: MAPREDUCE-1915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1915 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0, 0.20.2, 0.20.1, 0.20.3, 0.22.0 >Reporter: Rares Vernica >Assignee: Priyo Mustafi >Priority: Trivial > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1915.patch, MAPREDUCE-1915_1.patch > > > When checking if the "reduce" index is out of bounds the check should be > info.mapSpillRecord.size() <= reduce > instead of: > info.mapSpillRecord.size() < reduce > Not a big bug since an Out Of Bounds is thrown downstream anyway. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1915) IndexCache - getIndexInformation - check reduce index Out Of Bounds
[ https://issues.apache.org/jira/browse/MAPREDUCE-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyo Mustafi reassigned MAPREDUCE-1915: Assignee: Priyo Mustafi > IndexCache - getIndexInformation - check reduce index Out Of Bounds > --- > > Key: MAPREDUCE-1915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1915 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0 >Reporter: Rares Vernica >Assignee: Priyo Mustafi >Priority: Trivial > Attachments: MAPREDUCE-1915.patch, MAPREDUCE-1915_1.patch > > > When checking if the "reduce" index is out of bounds the check should be > info.mapSpillRecord.size() <= reduce > instead of: > info.mapSpillRecord.size() < reduce > Not a big bug since an Out Of Bounds is thrown downstream anyway. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1915) IndexCache - getIndexInformation - check reduce index Out Of Bounds
[ https://issues.apache.org/jira/browse/MAPREDUCE-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyo Mustafi updated MAPREDUCE-1915: - Status: Open (was: Patch Available) > IndexCache - getIndexInformation - check reduce index Out Of Bounds > --- > > Key: MAPREDUCE-1915 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1915 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0, 0.20.2, 0.20.1, 0.20.3, 0.22.0 >Reporter: Rares Vernica >Assignee: Priyo Mustafi >Priority: Trivial > Attachments: MAPREDUCE-1915.patch, MAPREDUCE-1915_1.patch > > > When checking if the "reduce" index is out of bounds the check should be > info.mapSpillRecord.size() <= reduce > instead of: > info.mapSpillRecord.size() < reduce > Not a big bug since an Out Of Bounds is thrown downstream anyway. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1754) Replace mapred.persmissions.supergroup with an acl : mapreduce.cluster.administrators
[ https://issues.apache.org/jira/browse/MAPREDUCE-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988259#action_12988259 ] Hudson commented on MAPREDUCE-1754: --- Integrated in Hadoop-Mapreduce-trunk-Commit #589 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/589/]) MAPREDUCE-1754. Replace mapred.persmissions.supergroup with an acl : mapreduce.cluster.administrators. Contributed by Amareshwari Sriramadasu. > Replace mapred.persmissions.supergroup with an acl : > mapreduce.cluster.administrators > - > > Key: MAPREDUCE-1754 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1754 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1754-1.txt, patch-1754-1.txt, > patch-1754-ydist.txt, patch-1754.txt > > > mapred.permissions.supergroup should be replaced with an acl so that it does > not restrict the admins to a single group. > See more details on MAPREDUCE-1542. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1825) jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps and finishedReduces when job is not initialized
[ https://issues.apache.org/jira/browse/MAPREDUCE-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988252#action_12988252 ] Priyo Mustafi commented on MAPREDUCE-1825: -- Hi Scott and Konstantin, I tested by putting a breakpoint in JobInProgress.initTasks(). 1) jobqueue_details.jsp is working fine i.e. not locking up 2) FairSchedulerServlet is locking up in showPools method as soon as it tries to synchronize on "scheduler". It continues again as soon as JIB.initTasks() finish. Not sure how this is happening as initTasks() lock JIB's monitor and showPools lock scheduler's monitor. Anyway, the patch doesn't seem to address the FSS issue. > jobqueue_details.jsp and FairSchedulerServelet should not call finishedMaps > and finishedReduces when job is not initialized > --- > > Key: MAPREDUCE-1825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1825 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1825.txt, MAPREDUCE-1825_1.txt, > MAPREDUCE-1825_2.txt > > > JobInProgress.finishedMaps() and finishedReduces() are synchronized. They are > called from jobqueue_details.jsp and FairSchedulerServelet which iterates > through all jobs. If any job is in initialization, these pages don't come up > until the initialization finishes. > See > [comment|https://issues.apache.org/jira/browse/MAPREDUCE-1354?focusedCommentId=12834139&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12834139] > for more details -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1754) Replace mapred.persmissions.supergroup with an acl : mapreduce.cluster.administrators
[ https://issues.apache.org/jira/browse/MAPREDUCE-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-1754: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thank you Amareshwari. > Replace mapred.persmissions.supergroup with an acl : > mapreduce.cluster.administrators > - > > Key: MAPREDUCE-1754 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1754 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1754-1.txt, patch-1754-1.txt, > patch-1754-ydist.txt, patch-1754.txt > > > mapred.permissions.supergroup should be replaced with an acl so that it does > not restrict the admins to a single group. > See more details on MAPREDUCE-1542. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2242) LinuxTaskController doesn't properly escape environment variables
[
https://issues.apache.org/jira/browse/MAPREDUCE-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988246#action_12988246
]
Todd Lipcon commented on MAPREDUCE-2242:
Talked to Arun about this today - apparently this will be fixed as part of
MAPREDUCE-2178.
So we can ignore this for a bit if people are OK with tests failing in the
meantime.
> LinuxTaskController doesn't properly escape environment variables
> -
>
> Key: MAPREDUCE-2242
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2242
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task-controller
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Blocker
> Fix For: 0.22.0
>
> Attachments: mapreduce-2242.txt
>
>
> LinuxTaskController currently just writes "export FOO=bar" pairs into
> taskjvm.sh, which fails if the value has multiple words or contains a space.
> This is causing TestDebugScriptWithLinuxTaskController among others to fail
> on trunk with the following message:
> {code}
> export: 1: -Dhadoop.tasklog.iscleanup: bad variable name
> {code}
> since it generated a taskjvm.sh including the following:
> {code}
> export
> HADOOP_CLIENT_OPTS=-Dhadoop.tasklog.taskid=attempt_20110104180935141_0001_m_01_0
> -Dhadoop.tasklog.iscleanup=false -Dhadoop.tasklog.totalLogFileSize=0
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2289) Permissions race can make getStagingDir fail on local filesystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988233#action_12988233 ] Todd Lipcon commented on MAPREDUCE-2289: The central issue here is that the Java mkdir API doesn't take a mode, so whenever we create a directory, it starts out with umask-based permissions until we can call chmod on it. Seems to me it would be safe and fix the race if we had it *fix* permissions if they were wrong, rather than bailing out. Then the race would only cause a redundant chmod. Thoughts? > Permissions race can make getStagingDir fail on local filesystem > > > Key: MAPREDUCE-2289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > > I've observed the following race condition in TestFairSchedulerSystem which > uses a MiniMRCluster on top of RawLocalFileSystem: > - two threads call getStagingDir at the same time > - Thread A checks fs.exists(stagingArea) and sees false > -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS) > --- mkdirs calls the Java mkdir API which makes the file with umask-based > permissions > - Thread B runs, checks fs.exists(stagingArea) and sees true > -- checks permissions, sees the default permissions, and throws IOE > - Thread A resumes and sets correct permissions -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2289) Permissions race can make getStagingDir fail on local filesystem
Permissions race can make getStagingDir fail on local filesystem Key: MAPREDUCE-2289 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2289 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 I've observed the following race condition in TestFairSchedulerSystem which uses a MiniMRCluster on top of RawLocalFileSystem: - two threads call getStagingDir at the same time - Thread A checks fs.exists(stagingArea) and sees false -- Calls mkdirs(stagingArea, JOB_DIR_PERMISSIONS) --- mkdirs calls the Java mkdir API which makes the file with umask-based permissions - Thread B runs, checks fs.exists(stagingArea) and sees true -- checks permissions, sees the default permissions, and throws IOE - Thread A resumes and sets correct permissions -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2285) MiniMRCluster does not start after ant test-patch
[
https://issues.apache.org/jira/browse/MAPREDUCE-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988220#action_12988220
]
Ramkumar Vadali commented on MAPREDUCE-2285:
The patch fixes the problem. I am no ivy expert, but it looks good to me.
> MiniMRCluster does not start after ant test-patch
> -
>
> Key: MAPREDUCE-2285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Ramkumar Vadali
>Priority: Blocker
> Attachments: cp-bad, cp-good, fix-build.diff
>
>
> Any test using MiniMRCluster hangs in the MiniMRCluster constructor after
> running ant test-patch. Steps to reproduce:
> 1. ant -Dpatch.file= -Dforrest.home= forrest> -Dfindbugs.home= -Dscratch.dir=/tmp/testpatch
> -Djava5.home= test-patch
> 2. Run any test that creates MiniMRCluster, say ant test
> -Dtestcase=TestFileArgs (contrib/streaming)
> Expected result: Test should succeed
> Actual result: Test hangs in MiniMRCluster.. This does not happen if
> we run ant clean after ant test-patch
> Test output:
> {code}
> [junit] 11/01/27 12:11:43 INFO ipc.Server: IPC Server handler 3 on 58675:
> starting
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: TaskTracker up at:
> localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: Starting tracker
> tracker_host0.foo.com:localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:44 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 0 time(s).
> [junit] 11/01/27 12:11:45 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 1 time(s).
> [junit] 11/01/27 12:11:46 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 2 time(s).
> [junit] 11/01/27 12:11:47 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 3 time(s).
> [junit] 11/01/27 12:11:48 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 4 time(s).
> [junit] 11/01/27 12:11:49 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 5 time(s).
> [junit] 11/01/27 12:11:50 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 6 time(s).
> [junit] 11/01/27 12:11:51 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 7 time(s).
> [junit] 11/01/27 12:11:52 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 8 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 9 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.RPC: Server at localhost/127.0.0.1:0
> not available yet, Z...
> {code}
> Stack trace:
> {code}
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:611)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:429)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:504)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
> at org.apache.hadoop.ipc.Client.call(Client.java:1008)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
> at org.apache.hadoop.mapred.$Proxy11.getProtocolVersion(Unknown
> Source)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:235)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:275)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:206)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:185)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:169)
> at org.apache.hadoop.mapred.TaskTracker$2.run(TaskTracker.java:699)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1142)
> at
> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:695)
> - locked <0x7f3b8ccc3870> (a org.apache.hadoop.mapred.TaskTracker)
> at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1391)
> at
> org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.createTaskTracker(Min
[jira] Commented: (MAPREDUCE-2263) MapReduce side of HADOOP-6904
[ https://issues.apache.org/jira/browse/MAPREDUCE-2263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988174#action_12988174 ] dhruba borthakur commented on MAPREDUCE-2263: - +1, code looks good. > MapReduce side of HADOOP-6904 > - > > Key: MAPREDUCE-2263 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2263 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: mapreduceRPC.patch > > > Make changes in Map/Reduce to incorporate HADOOP-6904. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2287) add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles
[ https://issues.apache.org/jira/browse/MAPREDUCE-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-2287: -- Status: Patch Available (was: In Progress) refer to comment in HADOOP-7119 > add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT > web-consoles > -- > > Key: MAPREDUCE-2287 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2287 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: security >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ha-mapred.patch > > > This JIRA is for the MAPRED portion of HADOOP-7119 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2287) add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles
[ https://issues.apache.org/jira/browse/MAPREDUCE-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-2287: -- Attachment: ha-mapred.patch > add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT > web-consoles > -- > > Key: MAPREDUCE-2287 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2287 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: security >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ha-mapred.patch > > > This JIRA is for the MAPRED portion of HADOOP-7119 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2285) MiniMRCluster does not start after ant test-patch
[
https://issues.apache.org/jira/browse/MAPREDUCE-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated MAPREDUCE-2285:
---
Attachment: fix-build.diff
maybe something like this patch? Including **/*.jar from build/ivy/lib is going
to pull in all the jars from javadoc, releaseaudit, and jdiff, which apparently
have conflicting dependencies which cause the JT to crash and the tests to time
out.
After removing this we might be missing some dependencies which we'll have to
add to the specific projects ivy.xml files, but that's more correct anyhow.
> MiniMRCluster does not start after ant test-patch
> -
>
> Key: MAPREDUCE-2285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Ramkumar Vadali
>Priority: Blocker
> Attachments: cp-bad, cp-good, fix-build.diff
>
>
> Any test using MiniMRCluster hangs in the MiniMRCluster constructor after
> running ant test-patch. Steps to reproduce:
> 1. ant -Dpatch.file= -Dforrest.home= forrest> -Dfindbugs.home= -Dscratch.dir=/tmp/testpatch
> -Djava5.home= test-patch
> 2. Run any test that creates MiniMRCluster, say ant test
> -Dtestcase=TestFileArgs (contrib/streaming)
> Expected result: Test should succeed
> Actual result: Test hangs in MiniMRCluster.. This does not happen if
> we run ant clean after ant test-patch
> Test output:
> {code}
> [junit] 11/01/27 12:11:43 INFO ipc.Server: IPC Server handler 3 on 58675:
> starting
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: TaskTracker up at:
> localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: Starting tracker
> tracker_host0.foo.com:localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:44 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 0 time(s).
> [junit] 11/01/27 12:11:45 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 1 time(s).
> [junit] 11/01/27 12:11:46 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 2 time(s).
> [junit] 11/01/27 12:11:47 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 3 time(s).
> [junit] 11/01/27 12:11:48 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 4 time(s).
> [junit] 11/01/27 12:11:49 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 5 time(s).
> [junit] 11/01/27 12:11:50 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 6 time(s).
> [junit] 11/01/27 12:11:51 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 7 time(s).
> [junit] 11/01/27 12:11:52 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 8 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 9 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.RPC: Server at localhost/127.0.0.1:0
> not available yet, Z...
> {code}
> Stack trace:
> {code}
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:611)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:429)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:504)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
> at org.apache.hadoop.ipc.Client.call(Client.java:1008)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
> at org.apache.hadoop.mapred.$Proxy11.getProtocolVersion(Unknown
> Source)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:235)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:275)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:206)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:185)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:169)
> at org.apache.hadoop.mapred.TaskTracker$2.run(TaskTracker.java:699)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1142)
> at
> org.apache.hadoop.map
[jira] Updated: (MAPREDUCE-2285) MiniMRCluster does not start after ant test-patch
[
https://issues.apache.org/jira/browse/MAPREDUCE-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated MAPREDUCE-2285:
---
Attachment: cp-good
cp-bad
I think the difference is that, after we run test-patch,
build/ivy/lib/Hadoop/javadoc gets populated and on the classpath of the contrib
tests. Attached are the classpath for a bad invocation (post test-patch) vs a
good one (post clean)
Maybe we need to straighten out our ivy confs?
> MiniMRCluster does not start after ant test-patch
> -
>
> Key: MAPREDUCE-2285
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2285
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Ramkumar Vadali
>Priority: Blocker
> Attachments: cp-bad, cp-good
>
>
> Any test using MiniMRCluster hangs in the MiniMRCluster constructor after
> running ant test-patch. Steps to reproduce:
> 1. ant -Dpatch.file= -Dforrest.home= forrest> -Dfindbugs.home= -Dscratch.dir=/tmp/testpatch
> -Djava5.home= test-patch
> 2. Run any test that creates MiniMRCluster, say ant test
> -Dtestcase=TestFileArgs (contrib/streaming)
> Expected result: Test should succeed
> Actual result: Test hangs in MiniMRCluster.. This does not happen if
> we run ant clean after ant test-patch
> Test output:
> {code}
> [junit] 11/01/27 12:11:43 INFO ipc.Server: IPC Server handler 3 on 58675:
> starting
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: TaskTracker up at:
> localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:43 INFO mapred.TaskTracker: Starting tracker
> tracker_host0.foo.com:localhost.localdomain/127.0.0.1:58675
> [junit] 11/01/27 12:11:44 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 0 time(s).
> [junit] 11/01/27 12:11:45 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 1 time(s).
> [junit] 11/01/27 12:11:46 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 2 time(s).
> [junit] 11/01/27 12:11:47 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 3 time(s).
> [junit] 11/01/27 12:11:48 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 4 time(s).
> [junit] 11/01/27 12:11:49 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 5 time(s).
> [junit] 11/01/27 12:11:50 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 6 time(s).
> [junit] 11/01/27 12:11:51 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 7 time(s).
> [junit] 11/01/27 12:11:52 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 8 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:0. Already tried 9 time(s).
> [junit] 11/01/27 12:11:53 INFO ipc.RPC: Server at localhost/127.0.0.1:0
> not available yet, Z...
> {code}
> Stack trace:
> {code}
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:611)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:429)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:504)
> - locked <0x7f3b8dc08700> (a
> org.apache.hadoop.ipc.Client$Connection)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1164)
> at org.apache.hadoop.ipc.Client.call(Client.java:1008)
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
> at org.apache.hadoop.mapred.$Proxy11.getProtocolVersion(Unknown
> Source)
> at
> org.apache.hadoop.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:235)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:275)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:206)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:185)
> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:169)
> at org.apache.hadoop.mapred.TaskTracker$2.run(TaskTracker.java:699)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1142)
> at
> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:695)
> - locked <0x7f3b8ccc3870> (a org.apache.hadoop.mapred
