[jira] [Commented] (MAPREDUCE-4694) Inconsistency in reduce input record counters between the stable and evolving APIs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471509#comment-13471509 ] Chris McConnell commented on MAPREDUCE-4694: Seems like an interesting case. +1 to the new API and backport of the logic. Would seem to me that the counters should reflect the framework performance and general data flow, rather than the user specific application. Inconsistency in reduce input record counters between the stable and evolving APIs -- Key: MAPREDUCE-4694 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4694 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.0.0-alpha Reporter: Harsh J Assignee: Harsh J Labels: inconsistency, regression, test Attachments: MAPREDUCE-4694.patch, MAPREDUCE-4694.patch In the stable (mapred) API execution, if the values iterator is skipped by a user, the records underneath it aren't counted in the Reduce input records counter as the key progresses to the next unique one. In the evolving API (mapreduce) API execution, if the values iterator is skipped by a user, the records underneath it is still counted as the key progresses to the next unique one. This behavior comes to me as a faulty one in the old API. A Reduce input records counter must always define all the records that have been passed into a reducer (cause they are read regardless of skipping), and both API's record counting despite user applications must be consistent. I'll post a test case illustrating this shortly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: MAPREDUCE-4678.patch Patch version 2 -- sorry about that Harsh! Cheers! Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: (was: MAPREDUCE-4678.patch) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: (was: MAPREDUCE-4678.patch) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: MAPREDUCE-4678.patch Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch, MAPREDUCE-4678.patch.txt HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris McConnell updated MAPREDUCE-4678: --- Attachment: MAPREDUCE-4678.patch Patch v1 Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464237#comment-13464237 ] Chris McConnell commented on MAPREDUCE-4678: There doesn't seem to be a test class for this example, if we need/want one let me know. Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException --- Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0, 0.23.0 Reporter: Chris McConnell Priority: Minor Attachments: MAPREDUCE-4678.patch HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4678) Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
Chris McConnell created MAPREDUCE-4678: -- Summary: Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException Key: MAPREDUCE-4678 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4678 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 0.20.205.0 Reporter: Chris McConnell Priority: Minor HADOOP_HOME/hadoop-examples.jar pentomino output_dir will fail with the following error message: INFO util.NativeCodeLoader: Loaded the native-hadoop library INFO mapred.FileInputFormat: Total input paths to process : 1 INFO mapred.JobClient: Running job: job_x INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: Task Id : attempt_, Status : FAILED java.lang.NegativeArraySizeException at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:95) at org.apache.hadoop.examples.dancing.DistributedPentomino$PentMap.map(Di stributedPentomino.java:51) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformat ion.java:1177) at org.apache.hadoop.mapred.Child.main(Child.java:264) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
