[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2

2017-07-06 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077502#comment-16077502
 ] 

Hadoop QA commented on MAPREDUCE-6246:
--

| (/) *{color:green} 1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 13m 
10s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green}  1m 
11s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green}  1m 
40s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green}  2m 
38s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green}107m 
31s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch 
passed. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 41s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6246 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12875244/MAPREDUCE-6246.004.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 0e41911e4bd8 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 7576a68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7020/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7020/console |
| Powered by | Apache Yetus 0.4.0   http://yetus.apache.org |


This message was automatically generated.




[jira] [Commented] (MAPREDUCE-6246) DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2

2017-07-06 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077395#comment-16077395
 ] 

Hadoop QA commented on MAPREDUCE-6246:
--

| (/) *{color:green} 1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 
48s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 15m 
19s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green}  1m 
15s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green}  2m 
39s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. 
{color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green}105m  
3s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch 
passed. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6246 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12875244/MAPREDUCE-6246.004.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 5c7abcfa1d63 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 7576a68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7019/testReport/ |
| modules | C: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7019/console |
| Powered by | Apache Yetus 0.4.0   http://yetus.apache.org |


This message was automatically generated.




[jira] [Commented] (MAPREDUCE-4950) MR App Master fails to write the history due to AvroTypeException

2017-07-06 Thread Raul Gutierrez Segales (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077088#comment-16077088
 ] 

Raul Gutierrez Segales commented on MAPREDUCE-4950:
---

seeing this again with 2.7.1, trying to confirm if 
{{mapreduce.map.speculative=false}} makes it go away. 

> MR App Master fails to write the history due to AvroTypeException
> -
>
> Key: MAPREDUCE-4950
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4950
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mr-am
>Reporter: Devaraj K
>Priority: Critical
>
> {code:xml}
> 2013-01-19 19:31:27,269 INFO [AsyncDispatcher event handler] 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, 
> writing event MAP_ATTEMPT_STARTED
> 2013-01-19 19:31:27,269 INFO [AsyncDispatcher event handler] 
> org.apache.hadoop.yarn.service.CompositeService: Error stopping 
> JobHistoryEventHandler
> org.apache.avro.AvroTypeException: Attempt to process a enum when a 
> array-start was expected.
>   at org.apache.avro.io.parsing.Parser.advance(Parser.java:93)
>   at org.apache.avro.io.JsonEncoder.writeEnum(JsonEncoder.java:210)
>   at 
> org.apache.avro.specific.SpecificDatumWriter.writeEnum(SpecificDatumWriter.java:54)
>   at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
>   at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
>   at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>   at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:66)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$MetaInfo.writeEvent(JobHistoryEventHandler.java:825)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:517)
>   at 
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.stop(JobHistoryEventHandler.java:346)
>   at 
> org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
>   at 
> org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:445)
>   at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:406)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
>   at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
>   at java.lang.Thread.run(Thread.java:662)
> 2013-01-19 19:31:27,271 INFO [AsyncDispatcher event handler] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory 
> hdfs://hacluster /root/staging-dir/root/.staging/job_1358603069474_0135
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6885) JobHistory event handling does not complete if handling event throws exception on shutdown

2017-07-06 Thread Haibo Chen (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076903#comment-16076903
 ] 

Haibo Chen commented on MAPREDUCE-6885:
---

[~vrushalic] This is what I was referring to that may be related to the issue 
of no jhist file that you encountered.

> JobHistory event handling does not complete if handling event throws 
> exception on shutdown
> --
>
> Key: MAPREDUCE-6885
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6885
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Jonathan Hung
>
> If eventHandlingThread handles an event which causes it to throw an exception 
> (e.g. if it is unable to flush an event to HDFS), the thread dies. The events 
> are enqueued and eventually handled when JobHistoryEventHandler stops. If 
> handling these events also throws an exception, the remaining events are 
> lost. This can for example cause moving job history files to 
> mapreduce.jobhistory.done-dir to not occur.
> There should be some fail-proof logic here to prevent these events from being 
> lost. Should also be careful that the same exception is not thrown for each 
> event to prevent the logs from being cluttered with the same stacktrace. 
> Perhaps we can set a configurable number of failed handleEvent calls before 
> finally giving up a clean shutdown.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Updated] (MAPREDUCE-6911) TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails consistently

2017-07-06 Thread Jason Lowe (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-6911:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.8.2
   2.9.0
   Status: Resolved  (was: Patch Available)

Thanks, Eric!  I committed this to branch-2, branch-2.8, and branch-2.8.2.

> TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails 
> consistently
> 
>
> Key: MAPREDUCE-6911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6911
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.9.0, 2.8.2
>Reporter: Eric Badger
>Assignee: Eric Badger
> Fix For: 2.9.0, 2.8.2
>
> Attachments: MAPREDUCE-6911.001.patch, 
> MAPREDUCE-6911-branch-2.001.patch
>
>
> {noformat}
> mapred-default.xml has 2 properties missing in  interface 
> org.apache.hadoop.mapreduce.MRJobConfig  interface 
> org.apache.hadoop.mapreduce.MRConfig  class 
> org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig  class 
> org.apache.hadoop.mapred.ShuffleHandler  class 
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat  class 
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat  class 
> org.apache.hadoop.mapreduce.Job  class 
> org.apache.hadoop.mapreduce.lib.input.NLineInputFormat  class 
> org.apache.hadoop.mapred.JobConf  class 
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
>   mapreduce.jobtracker.system.dir
>   mapreduce.jobtracker.staging.root.dir
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6911) TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails consistently

2017-07-06 Thread Jason Lowe (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076581#comment-16076581
 ] 

Jason Lowe commented on MAPREDUCE-6911:
---

+1 lgtm.  Committing this.

> TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails 
> consistently
> 
>
> Key: MAPREDUCE-6911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6911
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.9.0, 2.8.2
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6911.001.patch, 
> MAPREDUCE-6911-branch-2.001.patch
>
>
> {noformat}
> mapred-default.xml has 2 properties missing in  interface 
> org.apache.hadoop.mapreduce.MRJobConfig  interface 
> org.apache.hadoop.mapreduce.MRConfig  class 
> org.apache.hadoop.mapreduce.v2.jobhistory.JHAdminConfig  class 
> org.apache.hadoop.mapred.ShuffleHandler  class 
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat  class 
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat  class 
> org.apache.hadoop.mapreduce.Job  class 
> org.apache.hadoop.mapreduce.lib.input.NLineInputFormat  class 
> org.apache.hadoop.mapred.JobConf  class 
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
>   mapreduce.jobtracker.system.dir
>   mapreduce.jobtracker.staging.root.dir
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file

2017-07-06 Thread Peter Bacsko (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076441#comment-16076441
 ] 

Peter Bacsko commented on MAPREDUCE-6892:
-

Regarding JobUnsuccessfulCompletionEvent - we can certainly rename the fields. 
But the problem is that the "finishedMaps" and "finishedReduces" are also 
defined in the Avro schema (Events.avpr). Shall we rename those too? I believe 
that would introduce a big backward incompatible change. 

I also found inconsistent naming in JobFinishedEvent. Right now it looks like 
this:

{code}
tEvent.addInfo("NUM_MAPS", getFinishedMaps());
tEvent.addInfo("NUM_REDUCES", getFinishedReduces());
tEvent.addInfo("FAILED_MAPS", getFailedMaps());
tEvent.addInfo("FAILED_REDUCES", getFailedReduces());
tEvent.addInfo("FINISHED_MAPS", getFinishedMaps());
tEvent.addInfo("FINISHED_REDUCES", getFinishedReduces());
{code}

So NUM_* and FINISHED_* keys contain the same thing.

> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH
>
>
> Recently we encountered some issues with the value of failed tasks. After 
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually 
> there were failures. 
> Another minor thing is that you cannot get the number of killed tasks 
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the 
> successful map/reduce task counts. Number of failed (or killed) tasks are not 
> stored.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org