[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1813: - Status: Resolved (was: Patch Available) Release Note: Fixed an NPE in streaming that occurs when there is no input to reduce and the streaming reducer sends status updates by writing "reporter:status: xxx" statements to stderr. Resolution: Fixed I just committed this to trunk. Thanks Ravi! > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.3.patch, > 1813.v1.4.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1813: - Status: Patch Available (was: Open) Hadoop Flags: [Reviewed] +1 for the patch. Submitting to Hudson. > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.3.patch, > 1813.v1.4.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1813: Attachment: 1813.v1.4.patch Attaching new patch as I missed replacing one occurence of readOutput() call in earlier patch. > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.3.patch, > 1813.v1.4.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1813: Attachment: 1813.v1.3.patch Attaching new patch with the testcases refactoring. Moved the validation of task status from testStreamingStatus() to testReporting() and removed testStreamingStatus() method. Modified the perl script of TestStreamingStatus.java to write to stdout(similar to that is there in TestStreamingEmptyInpnonemptyOut.java) and removed the file TestStreamingEmptyInpnonemptyOut.java. As there are 3 copies of the method readOutput() in different files, removed 2 of the 3 readOutput() methods and kept the one in MapReduceTestUtil.java. > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.3.patch, > 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1813: Status: Open (was: Patch Available) > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1813: --- Fix Version/s: 0.22.0 (was: 0.20.3) > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1813: --- Status: Patch Available (was: Open) > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.20.3 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1813: Attachment: 1813.v1.2.patch Attaching patch by changing the testcases to have {perl script with empty input having "reporter:status:" lines and "reporter:counter:" lines written to stderr} instead of {StderrApp.class being used as streaming task}. Because, with nonempty input, earlier patch's testcase was not causing NPE consistently(fails based on timing) without the fix of the patch. Please review and provide your comments. > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.20.3 > > Attachments: 1813.patch, 1813.v1.2.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1813: Status: Open (was: Patch Available) > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.20.3 > > Attachments: 1813.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1813: Attachment: 1813.v1.patch Attaching patch that delays the starting of error thread till output thread is started so that we don't lose any stderr "report:status:" lines because of dummy reporter in case of having "report:status:" lines before consuming input in the reducer. > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.20.3 > > Attachments: 1813.patch, 1813.v1.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1813: --- Status: Patch Available (was: Open) > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.20.3 > > Attachments: 1813.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
[ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1813: Attachment: 1813.patch Attaching patch for trunk fixing the issue. This patch sets the reporter of MRErrorThread(in PipeMapRed.java) to a dummy reporter at the beginning of MRErrorThread.run() method just to avoid NPE. The actual reporter will be later set properly as usual and reports as needed. In the interim(before the actual reporter is set), the dummy reporter will just ignore the reporting lines. This is the way MRErrorThread's reporter is set to dummy reporter in waitOutputThreads() for the case of empty input based reducer. See HADOOP-4620 for details of the same. But before control goes to waitOutputThreads(), if the MRErrorThread's reporter is accessed, we get NPE. So to avoid this NPE, am setting this dummy reporter. This patch adds testcases which tests the perl script given in earlier comment as mapper and as reducer. Both test cases (1) perl script as streaming job's mapper and (2) perl script as streaming based reducer fail without the fix and pass with the fix. Please review and provide your comments. > NPE in PipeMapred.MRErrorThread > --- > > Key: MAPREDUCE-1813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.20.3 > > Attachments: 1813.patch > > > Some reduce tasks fail with following NPE > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325) > at > org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540) > at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137) > at > org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > Caused by: java.lang.NullPointerException >at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517) > at > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.