[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077502#comment-16077502 ] Hadoop QA commented on MAPREDUCE-6246: -- | (/) *{color:green} 1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 13m 10s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m 41s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 1m 11s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javac {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} unit {color} | {color:green} 2m 38s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green} 1{color} | {color:green} unit {color} | {color:green}107m 31s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. {color} | | {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}136m 41s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | MAPREDUCE-6246 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12875244/MAPREDUCE-6246.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 0e41911e4bd8 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 7576a68 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7020/testReport/ | | modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client | | Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7020/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated.
[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077395#comment-16077395 ] Hadoop QA commented on MAPREDUCE-6246: -- | (/) *{color:green} 1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 48s{color} | {color:blue} Docker mode activated. {color} | | {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 15m 19s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m 41s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javac {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} unit {color} | {color:green} 2m 39s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green} 1{color} | {color:green} unit {color} | {color:green}105m 3s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed. {color} | | {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}156m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | MAPREDUCE-6246 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12875244/MAPREDUCE-6246.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 5c7abcfa1d63 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 7576a68 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7019/testReport/ | | modules | C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client | | Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7019/console | | Powered by | Apache Yetus 0.4.0 http://yetus.apache.org | This message was automatically generated.
[jira] [Commented] (MAPREDUCE-4950) MR App Master fails to write the history due to AvroTypeException
[ https://issues.apache.org/jira/browse/MAPREDUCE-4950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077088#comment-16077088 ] Raul Gutierrez Segales commented on MAPREDUCE-4950: --- seeing this again with 2.7.1, trying to confirm if {{mapreduce.map.speculative=false}} makes it go away. > MR App Master fails to write the history due to AvroTypeException > - > > Key: MAPREDUCE-4950 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4950 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mr-am >Reporter: Devaraj K >Priority: Critical > > {code:xml} > 2013-01-19 19:31:27,269 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, > writing event MAP_ATTEMPT_STARTED > 2013-01-19 19:31:27,269 INFO [AsyncDispatcher event handler] > org.apache.hadoop.yarn.service.CompositeService: Error stopping > JobHistoryEventHandler > org.apache.avro.AvroTypeException: Attempt to process a enum when a > array-start was expected. > at org.apache.avro.io.parsing.Parser.advance(Parser.java:93) > at org.apache.avro.io.JsonEncoder.writeEnum(JsonEncoder.java:210) > at > org.apache.avro.specific.SpecificDatumWriter.writeEnum(SpecificDatumWriter.java:54) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66) > at > org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65) > at > org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57) > at > org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:66) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$MetaInfo.writeEvent(JobHistoryEventHandler.java:825) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:517) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.stop(JobHistoryEventHandler.java:346) > at > org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) > at > org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:445) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:406) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75) > at java.lang.Thread.run(Thread.java:662) > 2013-01-19 19:31:27,271 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory > hdfs://hacluster /root/staging-dir/root/.staging/job_1358603069474_0135 > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6885) JobHistory event handling does not complete if handling event throws exception on shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076903#comment-16076903 ] Haibo Chen commented on MAPREDUCE-6885: --- [~vrushalic] This is what I was referring to that may be related to the issue of no jhist file that you encountered. > JobHistory event handling does not complete if handling event throws > exception on shutdown > -- > > Key: MAPREDUCE-6885 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6885 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Hung > > If eventHandlingThread handles an event which causes it to throw an exception > (e.g. if it is unable to flush an event to HDFS), the thread dies. The events > are enqueued and eventually handled when JobHistoryEventHandler stops. If > handling these events also throws an exception, the remaining events are > lost. This can for example cause moving job history files to > mapreduce.jobhistory.done-dir to not occur. > There should be some fail-proof logic here to prevent these events from being > lost. Should also be careful that the same exception is not thrown for each > event to prevent the logs from being cluttered with the same stacktrace. > Perhaps we can set a configurable number of failed handleEvent calls before > finally giving up a clean shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6911) TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails consistently
[ https://issues.apache.org/jira/browse/MAPREDUCE-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6911: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.2 2.9.0 Status: Resolved (was: Patch Available) Thanks, Eric! I committed this to branch-2, branch-2.8, and branch-2.8.2. > TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails > consistently > > > Key: MAPREDUCE-6911 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6911 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.9.0, 2.8.2 >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 2.9.0, 2.8.2 > > Attachments: MAPREDUCE-6911.001.patch, > MAPREDUCE-6911-branch-2.001.patch > > > {noformat} > mapred-default.xml has 2 properties missing in interface > org.apache.hadoop.mapreduce.MRJobConfig interface > org.apache.hadoop.mapreduce.MRConfig class > org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig class > org.apache.hadoop.mapred.ShuffleHandler class > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat class > org.apache.hadoop.mapreduce.lib.input.FileInputFormat class > org.apache.hadoop.mapreduce.Job class > org.apache.hadoop.mapreduce.lib.input.NLineInputFormat class > org.apache.hadoop.mapred.JobConf class > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter > mapreduce.jobtracker.system.dir > mapreduce.jobtracker.staging.root.dir > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6911) TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails consistently
[ https://issues.apache.org/jira/browse/MAPREDUCE-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076581#comment-16076581 ] Jason Lowe commented on MAPREDUCE-6911: --- +1 lgtm. Committing this. > TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails > consistently > > > Key: MAPREDUCE-6911 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6911 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.9.0, 2.8.2 >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: MAPREDUCE-6911.001.patch, > MAPREDUCE-6911-branch-2.001.patch > > > {noformat} > mapred-default.xml has 2 properties missing in interface > org.apache.hadoop.mapreduce.MRJobConfig interface > org.apache.hadoop.mapreduce.MRConfig class > org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig class > org.apache.hadoop.mapred.ShuffleHandler class > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat class > org.apache.hadoop.mapreduce.lib.input.FileInputFormat class > org.apache.hadoop.mapreduce.Job class > org.apache.hadoop.mapreduce.lib.input.NLineInputFormat class > org.apache.hadoop.mapred.JobConf class > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter > mapreduce.jobtracker.system.dir > mapreduce.jobtracker.staging.root.dir > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[ https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076441#comment-16076441 ] Peter Bacsko commented on MAPREDUCE-6892: - Regarding JobUnsuccessfulCompletionEvent - we can certainly rename the fields. But the problem is that the "finishedMaps" and "finishedReduces" are also defined in the Avro schema (Events.avpr). Shall we rename those too? I believe that would introduce a big backward incompatible change. I also found inconsistent naming in JobFinishedEvent. Right now it looks like this: {code} tEvent.addInfo("NUM_MAPS", getFinishedMaps()); tEvent.addInfo("NUM_REDUCES", getFinishedReduces()); tEvent.addInfo("FAILED_MAPS", getFailedMaps()); tEvent.addInfo("FAILED_REDUCES", getFailedReduces()); tEvent.addInfo("FINISHED_MAPS", getFinishedMaps()); tEvent.addInfo("FINISHED_REDUCES", getFinishedReduces()); {code} So NUM_* and FINISHED_* keys contain the same thing. > Issues with the count of failed/killed tasks in the jhist file > -- > > Key: MAPREDUCE-6892 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, jobhistoryserver >Reporter: Peter Bacsko >Assignee: Peter Bacsko > Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH > > > Recently we encountered some issues with the value of failed tasks. After > parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually > there were failures. > Another minor thing is that you cannot get the number of killed tasks > (although this can be calculated). > The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the > successful map/reduce task counts. Number of failed (or killed) tasks are not > stored. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org