[jira] [Updated] (MAPREDUCE-109) Setting up ctr-A as custom delimiter for "mapred.textoutputformat.separator"
[
https://issues.apache.org/jira/browse/MAPREDUCE-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Egner updated MAPREDUCE-109:
Attachment: MAPREDUCE-109.patch
Re-uploading current version of patch. Sorry.
> Setting up ctr-A as custom delimiter for "mapred.textoutputformat.separator"
>
>
> Key: MAPREDUCE-109
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-109
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.20.2, 0.23.0
>Reporter: Suhas Gogate
> Attachments: MAPREDUCE-109.patch, MAPREDUCE-109.patch,
> MAPREDUCE-109.patch, MAPREDUCE-109.patch
>
>
> Feature added by this Jira has a problem while setting up some of the invalid
> xml characters e.g. ctrl-A e.g. mapred.textoutputformat.separator = "\u0001"
> e,g,
> String delim = "\u0001";
> Conf.set("mapred.textoutputformat.separator", delim);
> Job client serializes the jobconf with mapred.textoutputformat.separator set
> to "\u0001" (ctrl-A) and problem happens when it is de-serialized (read back)
> by job tracker, where it encounters invalid xml character.
> The test for this feature public : testFormatWithCustomSeparator() does not
> serialize the jobconf after adding the separator as ctrl-A and hence does not
> detect the specific problem.
> Here is an exception:
> 08/12/06 01:40:50 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.RuntimeException: org.xml.sax.SAXParseException: Character
> reference "" is an invalid XML
> character.
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:961)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:864)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:832)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:291)
> at
> org.apache.hadoop.mapred.JobConf.getJobPriority(JobConf.java:1163)
> at
> org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:179)
> at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1783)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> at org.apache.hadoop.ipc.Client.call(Client.java:715)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2472) Extra whitespace in mapred.child.java.opts breaks JVM initialization
Extra whitespace in mapred.child.java.opts breaks JVM initialization Key: MAPREDUCE-2472 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2472 Project: Hadoop Map/Reduce Issue Type: Bug Components: task-controller Affects Versions: 0.20.2, 0.22.0 Reporter: Todd Lipcon Fix For: 0.22.0 When creating taskjvm.sh, we split mapred.child.java.opts on " " and then create a quoted argument for each of those results. So, if you have an extra space anywhere in this configuration, you get an argument '' in the child command line, which the JVM interprets as an empty class name. This results in a ClassNotFoundException and the task cannot run. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1280) Eclipse Plugin does not work with Eclipse Ganymede (3.4)
[ https://issues.apache.org/jira/browse/MAPREDUCE-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-1280: - Fix Version/s: 0.20.203.0 > Eclipse Plugin does not work with Eclipse Ganymede (3.4) > > > Key: MAPREDUCE-1280 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1280 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.1, 0.21.1, 0.22.0 > Environment: Windows XP with Eclipse "Ganymede" 3.4 >Reporter: Aaron Kimball >Assignee: Alex Kozlov > Fix For: 0.20.3, 0.20.203.0, 0.21.1, 0.22.0 > > Attachments: HADOOP-3744.patch, HadoopApplicationLaunchShortcut.diff, > alex-HADOOP-3744.patch, hadoop-eclipse-plugin-0.20.3-SNAPSHOT.jar > > > The newest version of Eclipse seems incompatible with the plugin. The plugin > as released in 0.16.4 will allow you to add/remove MapReduce servers, and > will allow you to browse/manipulate the DFS in the DFS Browser, but will not > allow you to run programs. Clicking "Run As * Run On Hadoop" will simply not > cause the run-on-hadoop server selection window to appear. No error message > is given. > Dropping the 0.17.1 copy of the plugin JAR into the eclipse/plugins/ > directory does not fix the issue; it is in fact worse: Eclipse does not seem > to regard the 0.17 plugin as real. No "MapReduce Perspective" is made > available in the perspectives selection window. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-109) Setting up ctr-A as custom delimiter for "mapred.textoutputformat.separator"
[
https://issues.apache.org/jira/browse/MAPREDUCE-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Egner updated MAPREDUCE-109:
Attachment: MAPREDUCE-109.patch
Added test cases.
> Setting up ctr-A as custom delimiter for "mapred.textoutputformat.separator"
>
>
> Key: MAPREDUCE-109
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-109
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.20.2, 0.23.0
>Reporter: Suhas Gogate
> Attachments: MAPREDUCE-109.patch, MAPREDUCE-109.patch,
> MAPREDUCE-109.patch
>
>
> Feature added by this Jira has a problem while setting up some of the invalid
> xml characters e.g. ctrl-A e.g. mapred.textoutputformat.separator = "\u0001"
> e,g,
> String delim = "\u0001";
> Conf.set("mapred.textoutputformat.separator", delim);
> Job client serializes the jobconf with mapred.textoutputformat.separator set
> to "\u0001" (ctrl-A) and problem happens when it is de-serialized (read back)
> by job tracker, where it encounters invalid xml character.
> The test for this feature public : testFormatWithCustomSeparator() does not
> serialize the jobconf after adding the separator as ctrl-A and hence does not
> detect the specific problem.
> Here is an exception:
> 08/12/06 01:40:50 INFO mapred.FileInputFormat: Total input paths to process :
> 1
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.RuntimeException: org.xml.sax.SAXParseException: Character
> reference "" is an invalid XML
> character.
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:961)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:864)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:832)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:291)
> at
> org.apache.hadoop.mapred.JobConf.getJobPriority(JobConf.java:1163)
> at
> org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:179)
> at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1783)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> at org.apache.hadoop.ipc.Client.call(Client.java:715)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2471) MapReduce Math Library
[ https://issues.apache.org/jira/browse/MAPREDUCE-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028631#comment-13028631 ] Todd Lipcon commented on MAPREDUCE-2471: Does this belong in MR proper? Seems like a great github project or incubator proposal. > MapReduce Math Library > -- > > Key: MAPREDUCE-2471 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2471 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > > This is an umbrella JIRA for a MapReduce math library. > The core algorithms are _MapReduce-Sum_, _MapReduce-FFT_, > _MapReduce-Multiplication_, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028623#comment-13028623 ] Aaron Baff commented on MAPREDUCE-2470: --- This appears to occur only when the Job has been retired, and the JobInfo data has been removed from HDFS (default dir is /jobtracker/jobInfo I believe). In that case, apparently the getJobCounters() call returns NULL, instead of a Counters object with no Counter's in it. Thus, in the Counters.downgrade() for the old MR API, it doesn't check for NULL, instead just uses the results in a for-each loop. > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff > Attachments: counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2471) MapReduce Math Library
[ https://issues.apache.org/jira/browse/MAPREDUCE-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028550#comment-13028550 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-2471: --- Sorry, the link to the preprint PDF should be this: http://people.apache.org/~szetszwo/ssmr20110430.pdf > MapReduce Math Library > -- > > Key: MAPREDUCE-2471 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2471 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > > This is an umbrella JIRA for a MapReduce math library. > The core algorithms are _MapReduce-Sum_, _MapReduce-FFT_, > _MapReduce-Multiplication_, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2471) MapReduce Math Library
[ https://issues.apache.org/jira/browse/MAPREDUCE-2471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028549#comment-13028549 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-2471: --- I already have designed algorithms for computing summations, discrete Fourier transforms and multiplications on MapReduce; see [1]. Will post the implementation soon. [1] *Schönhage-Strassen Algorithm with MapReduce for Multiplying Terabit Integers* SNC 2011, to appear. ([Preprint PDF|http://www.cargo.wlu.ca/SNC2011/papers.html]) > MapReduce Math Library > -- > > Key: MAPREDUCE-2471 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2471 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > > This is an umbrella JIRA for a MapReduce math library. > The core algorithms are _MapReduce-Sum_, _MapReduce-FFT_, > _MapReduce-Multiplication_, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2471) MapReduce Math Library
MapReduce Math Library -- Key: MAPREDUCE-2471 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2471 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE This is an umbrella JIRA for a MapReduce math library. The core algorithms are _MapReduce-Sum_, _MapReduce-FFT_, _MapReduce-Multiplication_, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
[ https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Baff updated MAPREDUCE-2470: -- Attachment: counters_null_data.pcap Wireshark capture of RunningJob.getCounters() call. > Receiving NPE occasionally on RunningJob.getCounters() call > --- > > Key: MAPREDUCE-2470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.21.0 > Environment: FreeBSD, Java6, Hadoop r0.21.0 >Reporter: Aaron Baff > Attachments: counters_null_data.pcap > > > This is running in a Java daemon that is used as an interface (Thrift) to get > information and data from MR Jobs. Using JobClient.getJob(JobID) I > successfully get a RunningJob object (I'm checking for NULL), and then rarely > I get an NPE when I do RunningJob.getCounters(). This seems to occur after > the daemon has been up and running for a while, and in the event of an > Exception, I close the JobClient, set it to NULL, and a new one should then > be created on the next request for data. Yet, I still seem to be unable to > fetch the Counters. Below is the stack trace. > java.lang.NullPointerException > at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) > at > com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) > at > com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) > at > org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) > at > org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2467) HDFS-1052 changes break the raid contrib module in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-2467: --- Attachment: MR-2467.2.patch Updated patch with change from MAPREDUCE-2465 reverted. > HDFS-1052 changes break the raid contrib module in MapReduce > > > Key: MAPREDUCE-2467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: MR-2467.1.patch, MR-2467.2.patch, MR-2467.patch > > > Raid contrib module requires changes to work with the federation changes made > in HDFS-1052. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2467) HDFS-1052 changes break the raid contrib module in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-2467: --- Status: Patch Available (was: Open) > HDFS-1052 changes break the raid contrib module in MapReduce > > > Key: MAPREDUCE-2467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: MR-2467.1.patch, MR-2467.2.patch, MR-2467.patch > > > Raid contrib module requires changes to work with the federation changes made > in HDFS-1052. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2467) HDFS-1052 changes break the raid contrib module in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-2467: --- Status: Open (was: Patch Available) > HDFS-1052 changes break the raid contrib module in MapReduce > > > Key: MAPREDUCE-2467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: MR-2467.1.patch, MR-2467.patch > > > Raid contrib module requires changes to work with the federation changes made > in HDFS-1052. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2465) HDFS raid not compiling after federation merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028504#comment-13028504 ] Ramkumar Vadali commented on MAPREDUCE-2465: Suresh, the patch for MAPREDUCE-2467 looks good. > HDFS raid not compiling after federation merge > -- > > Key: MAPREDUCE-2465 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2465 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ramkumar Vadali >Priority: Blocker > Attachments: disable-raid-compilation.txt, failure.txt, > fix-compile-but-raid-broken.txt > > > The RAID contrib is no longer compiling now that federation has been merged, > due to some API changes in LocatedBlock and FSDataset. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2467) HDFS-1052 changes break the raid contrib module in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028503#comment-13028503 ] Ramkumar Vadali commented on MAPREDUCE-2467: Thanks for making the changes! > HDFS-1052 changes break the raid contrib module in MapReduce > > > Key: MAPREDUCE-2467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: MR-2467.1.patch, MR-2467.patch > > > Raid contrib module requires changes to work with the federation changes made > in HDFS-1052. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2467) HDFS-1052 changes break the raid contrib module in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028502#comment-13028502 ] Ramkumar Vadali commented on MAPREDUCE-2467: +1 looks good > HDFS-1052 changes break the raid contrib module in MapReduce > > > Key: MAPREDUCE-2467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: MR-2467.1.patch, MR-2467.patch > > > Raid contrib module requires changes to work with the federation changes made > in HDFS-1052. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2468) MR-279: Metrics for shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas resolved MAPREDUCE-2468. -- Resolution: Fixed Fix Version/s: (was: 0.23.0) Hadoop Flags: [Reviewed] +1 I committed this to the MR-279 branch. Thanks Luke! > MR-279: Metrics for shuffle > --- > > Key: MAPREDUCE-2468 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2468 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Luke Lu >Assignee: Luke Lu > Attachments: mr-2468-shuffle-metrics-v1.patch > > > Metrics for MR shuffle service. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call
Receiving NPE occasionally on RunningJob.getCounters() call --- Key: MAPREDUCE-2470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.21.0 Environment: FreeBSD, Java6, Hadoop r0.21.0 Reporter: Aaron Baff This is running in a Java daemon that is used as an interface (Thrift) to get information and data from MR Jobs. Using JobClient.getJob(JobID) I successfully get a RunningJob object (I'm checking for NULL), and then rarely I get an NPE when I do RunningJob.getCounters(). This seems to occur after the daemon has been up and running for a while, and in the event of an Exception, I close the JobClient, set it to NULL, and a new one should then be created on the next request for data. Yet, I still seem to be unable to fetch the Counters. Below is the stack trace. java.lang.NullPointerException at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77) at org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381) at com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545) at com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697) at org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-2429) Check jvmid during task status report
[ https://issues.apache.org/jira/browse/MAPREDUCE-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy reopened MAPREDUCE-2429: -- Sorry, my bad for closing this. Sid is working on porting this to trunk. Thanks for pointing it out. > Check jvmid during task status report > - > > Key: MAPREDUCE-2429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Arun C Murthy >Assignee: Siddharth Seth > Fix For: 0.20.204.0 > > Attachments: MR2429-1.patch, MR2429.patch > > > Currently TT doens't check to ensure jvmid is relevant during communication > with the Child via TaskUmbilicalProtocol. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2429) Check jvmid during task status report
[ https://issues.apache.org/jira/browse/MAPREDUCE-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028451#comment-13028451 ] Todd Lipcon commented on MAPREDUCE-2429: Does this not need to go into trunk? From looking at the code I see no reason that trunk wouldn't also be affected. > Check jvmid during task status report > - > > Key: MAPREDUCE-2429 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2429 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Arun C Murthy >Assignee: Siddharth Seth > Fix For: 0.20.204.0 > > Attachments: MR2429-1.patch, MR2429.patch > > > Currently TT doens't check to ensure jvmid is relevant during communication > with the Child via TaskUmbilicalProtocol. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2467) HDFS-1052 changes break the raid contrib module in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-2467: --- Status: Patch Available (was: Open) > HDFS-1052 changes break the raid contrib module in MapReduce > > > Key: MAPREDUCE-2467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: MR-2467.1.patch, MR-2467.patch > > > Raid contrib module requires changes to work with the federation changes made > in HDFS-1052. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2448) NoSuchMethodError: org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(..)
[
https://issues.apache.org/jira/browse/MAPREDUCE-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028209#comment-13028209
]
Hudson commented on MAPREDUCE-2448:
---
Integrated in Hadoop-Mapreduce-22-branch #44 (See
[https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/44/])
MAPREDUCE-2448. svn merge -c 1096520 from trunk
> NoSuchMethodError:
> org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(..)
> -
>
> Key: MAPREDUCE-2448
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2448
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid, test
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Eli Collins
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: mapreduce-2448-1.patch
>
>
> {noformat}
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(Ljava/lang/String;I)Z
> at
> org.apache.hadoop.raid.TestBlockFixer.corruptBlock(TestBlockFixer.java:643)
> at
> org.apache.hadoop.raid.TestBlockFixer.implBlockFix(TestBlockFixer.java:189)
> at
> org.apache.hadoop.raid.TestBlockFixer.testBlockFixLocal(TestBlockFixer.java:139)
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2457) job submission should inject group.name (on the JT side)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028210#comment-13028210 ] Hudson commented on MAPREDUCE-2457: --- Integrated in Hadoop-Mapreduce-22-branch #44 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/44/]) MAPREDUCE-2457. Job submission should inject group.name on the JobTracker. Contributed by Alejandro Abdelnur. > job submission should inject group.name (on the JT side) > > > Key: MAPREDUCE-2457 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2457 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.21.0, 0.22.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2457-1.patch, MAPREDUCE-2457.patch > > > Until Hadoop 0.20, the JobClient was injecting the property 'group.name' on > the JobConf submitted to the JobTracker. > Since Hadoop 0.21, due to security related changes, this is not done anymore. > This breaks backwards compatibility for jobs/components that expect the > 'group.name' to be automatically set at submission time. > An example of a component being affected by this change is the FairScheduler > where it is common to use the group.name as pool name. Different from other > properties, a special characteristic of the group.name is that its value > cannot be tampered by a user. > For security reasons this should not be done (as it was done before) in the > JobClient side. Instead, it should be done in the JobTracker when the JobConf > is received. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2445) TestMiniMRWithDFSWithDistinctUsers is very broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028211#comment-13028211 ] Hudson commented on MAPREDUCE-2445: --- Integrated in Hadoop-Mapreduce-22-branch #44 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/44/]) > TestMiniMRWithDFSWithDistinctUsers is very broken > - > > Key: MAPREDUCE-2445 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2445 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security, test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > Attachments: mapreduce-2445.txt, mapreduce-2445.txt > > > This test has a number of issues: > - it side steps the normal job submission API for no apparent reason, > manually writing splits file and uploading submission files. (but forgets to > upload the job jar, so the jobs all fail) > - it doesn't call waitForCompletion, or check job status (so it doesn't > notice that the jobs all fail) > - it doesn't verify in any way that the job output is owned by the user who > supposedly ran the job > - it shuts down DFS before MR > These all conspire to make it pass, but it isn't actually testing anything. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2467) HDFS-1052 changes break the raid contrib module in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028103#comment-13028103 ] Eli Collins commented on MAPREDUCE-2467: +1 lgtm > HDFS-1052 changes break the raid contrib module in MapReduce > > > Key: MAPREDUCE-2467 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2467 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: MR-2467.1.patch, MR-2467.patch > > > Raid contrib module requires changes to work with the federation changes made > in HDFS-1052. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
