[jira] [Updated] (MAPREDUCE-5894) Make critical YARN properties first class citizens in the build.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jay vyas updated MAPREDUCE-5894: Description: We recently found that when deploy hadoop 2.2 with hadoop 2.0 values {noformat} mapreduce_shuffle {noformat} changed to {noformat} mapreduce.shuffle {noformat} . There are likewise many similar examples of parameters which become deprecated over time. See http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html I suggest we: 1) Have a list of all mandatory *current* parameters stored in the code, and also, 2) a list of deprecated ones. 3) Then, have the build * automatically fail * if a parameter in the "madatory" list is NOT accessed during unit tests this would (a) make it so that unit testing of parameters does not regress and (b) force all updates to the code which change a parameter name, to also include an *update to deprecated parameter list*, before build passes. This setup could then be used to enable 4) Reporting or failing upon presence of clearly obsolete parameters which are result of obvious human error. was: We recently found that when deploy hadoop 2.2 with hadoop 2.0 values {noformat} mapreduce_shuffle {noformat} changed to {noformat} mapreduce.shuffle {noformat} . There are likewise many similar examples of parameters which become deprecated over time. See http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html I suggest we: 1) Have a list of all mandatory *current* parameters stored in the code, and also, 2) a list of deprecated ones. 3) Then, have the build * automatically fail * a parameter in the "madatory" list is NOT accessed. this would (a) make it so that unit testing of parameters does not regress and (b) force all updates to the code which change a parameter name, to also include update to deprecated parameter list, before build passes. > Make critical YARN properties first class citizens in the build. > > > Key: MAPREDUCE-5894 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5894 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: jay vyas > > We recently found that when deploy hadoop 2.2 with hadoop 2.0 values > {noformat} mapreduce_shuffle {noformat} changed to {noformat} > mapreduce.shuffle {noformat} . > There are likewise many similar examples of parameters which become > deprecated over time. See > http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html > I suggest we: > 1) Have a list of all mandatory *current* parameters stored in the code, and > also, > 2) a list of deprecated ones. > 3) Then, have the build * automatically fail * if a parameter in the > "madatory" list is NOT accessed during unit tests > this would (a) make it so that unit testing of parameters does not regress > and (b) force all updates to the code which change a parameter name, to also > include an *update to deprecated parameter list*, before build passes. > This setup could then be used to enable > 4) Reporting or failing upon presence of clearly obsolete parameters which > are result of obvious human error. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5894) Make critical YARN properties first class citizens in the build.
jay vyas created MAPREDUCE-5894: --- Summary: Make critical YARN properties first class citizens in the build. Key: MAPREDUCE-5894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5894 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: jay vyas We recently found that when deploy hadoop 2.2 with hadoop 2.0 values {noformat} mapreduce_shuffle {noformat} changed to {noformat} mapreduce.shuffle {noformat} . There are likewise many similar examples of parameters which become deprecated over time. See http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html I suggest we: 1) Use the *set of parameters which are deprecated* over time into java class which ships directly with the code, maybe even as a static list inside of Configuration() itself, with *optional extended parameters read from a configurable parameter *, so that ecosystem users (i.e. like Hbase, or alternative file systems) can add their own deprecation info. 2) have this list *checked on yarn daemon startup*. so that unused parameters which are *obviously artifacts are flagged immediately* by the daemon failing immediately. 3)Have a list of all mandatory *current* parameters stored in the code, and also, a list of deprecated ones. Then, have the build * automatically fail * a parameter in the "madatory" list is NOT accessed. this would (a) make it so that unit testing of parameters does not regress and (b) force all updates to the code which change a parameter name, to also include update to deprecated parameter list, before build passes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5894) Make critical YARN properties first class citizens in the build.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jay vyas updated MAPREDUCE-5894: Description: We recently found that when deploy hadoop 2.2 with hadoop 2.0 values {noformat} mapreduce_shuffle {noformat} changed to {noformat} mapreduce.shuffle {noformat} . There are likewise many similar examples of parameters which become deprecated over time. See http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html I suggest we: 1) Have a list of all mandatory *current* parameters stored in the code, and also, 2) a list of deprecated ones. 3) Then, have the build * automatically fail * a parameter in the "madatory" list is NOT accessed. this would (a) make it so that unit testing of parameters does not regress and (b) force all updates to the code which change a parameter name, to also include update to deprecated parameter list, before build passes. was: We recently found that when deploy hadoop 2.2 with hadoop 2.0 values {noformat} mapreduce_shuffle {noformat} changed to {noformat} mapreduce.shuffle {noformat} . There are likewise many similar examples of parameters which become deprecated over time. See http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html I suggest we: 1) Use the *set of parameters which are deprecated* over time into java class which ships directly with the code, maybe even as a static list inside of Configuration() itself, with *optional extended parameters read from a configurable parameter *, so that ecosystem users (i.e. like Hbase, or alternative file systems) can add their own deprecation info. 2) have this list *checked on yarn daemon startup*. so that unused parameters which are *obviously artifacts are flagged immediately* by the daemon failing immediately. 3)Have a list of all mandatory *current* parameters stored in the code, and also, a list of deprecated ones. Then, have the build * automatically fail * a parameter in the "madatory" list is NOT accessed. this would (a) make it so that unit testing of parameters does not regress and (b) force all updates to the code which change a parameter name, to also include update to deprecated parameter list, before build passes. > Make critical YARN properties first class citizens in the build. > > > Key: MAPREDUCE-5894 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5894 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: jay vyas > > We recently found that when deploy hadoop 2.2 with hadoop 2.0 values > {noformat} mapreduce_shuffle {noformat} changed to {noformat} > mapreduce.shuffle {noformat} . > There are likewise many similar examples of parameters which become > deprecated over time. See > http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html > I suggest we: > 1) Have a list of all mandatory *current* parameters stored in the code, and > also, > 2) a list of deprecated ones. > 3) Then, have the build * automatically fail * a parameter in the "madatory" > list is NOT accessed. this would (a) make it so that unit testing of > parameters does not regress and (b) force all updates to the code which > change a parameter name, to also include update to deprecated parameter list, > before build passes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5309) 2.0.4 JobHistoryParser can't parse certain failed job history files generated by 2.0.3 history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000909#comment-14000909 ] Hadoop QA commented on MAPREDUCE-5309: -- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645344/MAPREDUCE-5309-v4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4611//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4611//console This message is automatically generated. > 2.0.4 JobHistoryParser can't parse certain failed job history files generated > by 2.0.3 history server > - > > Key: MAPREDUCE-5309 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5309 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 2.0.4-alpha >Reporter: Vrushali C >Assignee: Rushabh S Shah > Attachments: MAPREDUCE-5309-v2.patch, MAPREDUCE-5309-v3.patch, > MAPREDUCE-5309-v4.patch, MAPREDUCE-5309.patch, Test20JobHistoryParsing.java, > job_2_0_3-KILLED.jhist > > > When the 2.0.4 JobHistoryParser tries to parse a job history file generated > by hadoop 2.0.3, the jobhistoryparser throws as an error as > java.lang.ClassCastException: org.apache.avro.generic.GenericData$Array > cannot be cast to org.apache.hadoop.mapreduce.jobhistory.JhCounters > at > org.apache.hadoop.mapreduce.jobhistory.TaskAttemptUnsuccessfulCompletion.put(TaskAttemptUnsuccessfulCompletion.java:58) > at org.apache.avro.generic.GenericData.setField(GenericData.java:463) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129) > at > org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:93) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:111) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:156) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:142) > at > com.twitter.somepackage.Test20JobHistoryParsing.testFileAvro(Test20JobHistoryParsing.java:23) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java: