[jira] [Updated] (MAPREDUCE-6499) Add elapsed time for retired job in JobHistoryServer WebUI

2015-11-13 Thread Akira AJISAKA (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-6499:
-
   Flags:   (was: Patch)
Priority: Major  (was: Trivial)
Hadoop Flags: Reviewed
Release Note:   (was: Add more informations of finished jobs on jobhistory 
main page.)
 Component/s: webapps
 Summary: Add elapsed time for retired job in JobHistoryServer WebUI  
(was: Add More Informations Of Jobs In JobHistory Main Page)

I'll commit this the day after tomorrow if there are no comments.

> Add elapsed time for retired job in JobHistoryServer WebUI
> --
>
> Key: MAPREDUCE-6499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: webapps
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HADOOP-MAPREDUCE-6499.2.patch, 
> HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, 
> HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.6.patch, 
> HADOOP-MAPREDUCE-6499.7.patch, HADOOP-MAPREDUCE-6499.8.patch, 
> HADOOP-MAPREDUCE-6499.9.patch, HADOOP-MAPREDUCE-6499.patch, Screen Shot 
> 2015-11-13 at 23.47.10.png
>
>
> Now in  JobHistory Main Page show too little information about finished 
> jobs,even don't have the job's running total time,only 
> startTime,finishedTime.So,in jobHistory main page,we can add more 
> informations about jobs, that we can better analyze jobs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (MAPREDUCE-5889) Deprecate FileInputFormat.setInputPaths(Job, String) and FileInputFormat.addInputPaths(Job, String)

2015-11-13 Thread Akira AJISAKA (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-5889:
-
Attachment: MAPREDUCE-5889.4.patch

Rebased for the latest trunk.

> Deprecate FileInputFormat.setInputPaths(Job, String) and 
> FileInputFormat.addInputPaths(Job, String)
> ---
>
> Key: MAPREDUCE-5889
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5889
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
>  Labels: BB2015-05-TBR, newbie
> Attachments: MAPREDUCE-5889.3.patch, MAPREDUCE-5889.4.patch, 
> MAPREDUCE-5889.patch, MAPREDUCE-5889.patch
>
>
> {{FileInputFormat.setInputPaths(Job job, String commaSeparatedPaths)}} and 
> {{FileInputFormat.addInputPaths(Job job, String commaSeparatedPaths)}} fail 
> to parse commaSeparatedPaths if a comma is included in the file path. (e.g. 
> Path: {{/path/file,with,comma}})
> We should deprecate these methods and document to use {{setInputPaths(Job 
> job, Path... inputPaths)}} and {{addInputPaths(Job job, Path... inputPaths)}} 
> instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (MAPREDUCE-6499) Add elapsed time for retired job in JobHistoryServer WebUI

2015-11-13 Thread Lin Yiqun (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004133#comment-15004133
 ] 

Lin Yiqun commented on MAPREDUCE-6499:
--

Thanks [~ajisakaa] reviews.

> Add elapsed time for retired job in JobHistoryServer WebUI
> --
>
> Key: MAPREDUCE-6499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: webapps
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: HADOOP-MAPREDUCE-6499.2.patch, 
> HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, 
> HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.6.patch, 
> HADOOP-MAPREDUCE-6499.7.patch, HADOOP-MAPREDUCE-6499.8.patch, 
> HADOOP-MAPREDUCE-6499.9.patch, HADOOP-MAPREDUCE-6499.patch, Screen Shot 
> 2015-11-13 at 23.47.10.png
>
>
> Now in  JobHistory Main Page show too little information about finished 
> jobs,even don't have the job's running total time,only 
> startTime,finishedTime.So,in jobHistory main page,we can add more 
> informations about jobs, that we can better analyze jobs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Reopened] (MAPREDUCE-3065) ApplicationMaster killed by NodeManager due to excessive virtual memory consumption

2015-11-13 Thread Vinod Kumar Vavilapalli (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli reopened MAPREDUCE-3065:


> ApplicationMaster killed by NodeManager due to excessive virtual memory 
> consumption
> ---
>
> Key: MAPREDUCE-3065
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3065
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.0.0-alpha
>Reporter: Chris Riccomini
>
> > Hey Vinod,
> > 
> > OK, so I have a little more clarity into this.
> > 
> > When I bump my resource request for my AM to 4096, it runs. The important 
> > line in the NM logs is:
> > 
> > 2011-09-21 13:43:44,366 INFO  monitor.ContainersMonitorImpl 
> > (ContainersMonitorImpl.java:run(402)) - Memory usage of ProcessTree 25656 
> > for container-id container_1316637655278_0001_01_01 : Virtual 
> > 2260938752 bytes, limit : 4294967296 bytes; Physical 120860672 bytes, limit 
> > -1 bytes
> > 
> > The thing to note is the virtual memory, which is off the charts, even 
> > though my physical memory is almost nothing (12 megs). I'm still poking 
> > around the code, but I am noticing that there are two checks in the NM, one 
> > for virtual mem, and one for physical mem. The virtual memory check appears 
> > to be toggle-able, but is presumably defaulted to on.
> > 
> > At this point I'm trying to figure out exactly what the VMEM check is for, 
> > why YARN thinks my app is taking 2 gigs, and how to fix this.
> > 
> > Cheers,
> > Chris
> > 
> > From: Chris Riccomini [criccom...@linkedin.com]
> > Sent: Wednesday, September 21, 2011 1:42 PM
> > To: mapreduce-...@hadoop.apache.org
> > Subject: Re: ApplicationMaster Memory Usage
> > 
> > For the record, I bumped to 4096 for memory resource request, and it works.
> > :(
> > 
> > 
> > On 9/21/11 1:32 PM, "Chris Riccomini"  wrote:
> > 
> >> Hey Vinod,
> >> 
> >> So, I ran my application master directly from the CLI. I commented out the
> >> YARN-specific code. It runs fine without leaking memory.
> >> 
> >> I then ran it from YARN, with all YARN-specific code commented it. It again
> >> ran fine.
> >> 
> >> I then uncommented JUST my registerWithResourceManager call. It then fails
> >> with OOM after a few seconds. I call registerWithResourceManager, and then 
> >> go
> >> into a while(true) { println("yeh") sleep(1000) }. Doing this prints:
> >> 
> >> yeh
> >> yeh
> >> yeh
> >> yeh
> >> yeh
> >> 
> >> At which point, it dies, and, in the NodeManager,I see:
> >> 
> >> 2011-09-21 13:24:51,036 WARN  monitor.ContainersMonitorImpl
> >> (ContainersMonitorImpl.java:isProcessTreeOverLimit(289)) - Process tree for
> >> container: container_1316626117280_0005_01_01 has processes older than 
> >> 1
> >> iteration running over the configured limit. Limit=2147483648, current 
> >> usage =
> >> 2192773120
> >> 2011-09-21 13:24:51,037 WARN  monitor.ContainersMonitorImpl
> >> (ContainersMonitorImpl.java:run(453)) - Container
> >> [pid=23852,containerID=container_1316626117280_0005_01_01] is running
> >> beyond memory-limits. Current usage : 2192773120bytes. Limit :
> >> 2147483648bytes. Killing container.
> >> Dump of the process-tree for container_1316626117280_0005_01_01 :
> >> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> >> SYSTEM_TIME(MILLIS)
> >> VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> >> |- 23852 20570 23852 23852 (bash) 0 0 108638208 303 /bin/bash -c java 
> >> -Xmx512M
> >> -cp './package/*' kafka.yarn.ApplicationMaster
> >> /home/criccomi/git/kafka-yarn/dist/kafka-streamer.tgz 5 1 1316626117280
> >> com.linkedin.TODO 1
> >> 1>/tmp/logs/application_1316626117280_0005/container_1316626117280_0005_01_000
> >> 001/stdout
> >> 2>/tmp/logs/application_1316626117280_0005/container_1316626117280_0005_01_000
> >> 001/stderr
> >> |- 23855 23852 23852 23852 (java) 81 4 2084134912 14772 java -Xmx512M -cp
> >> ./package/* kafka.yarn.ApplicationMaster
> >> /home/criccomi/git/kafka-yarn/dist/kafka-streamer.tgz 5 1 1316626117280
> >> com.linkedin.TODO 1
> >> 2011-09-21 13:24:51,037 INFO  monitor.ContainersMonitorImpl
> >> (ContainersMonitorImpl.java:run(463)) - Removed ProcessTree with root 23852
> >> 
> >> Either something is leaking in YARN, or my registerWithResourceManager code
> >> (see below) is doing something funky.
> >> 
> >> I'm trying to avoid going through all the pain of attaching a remote 
> >> debugger.
> >> Presumably things aren't leaking in YARN, which means it's likely that I'm
> >> doing something wrong in my registration code.
> >> 
> >> Incidentally, my NodeManager is running with 1000 megs. My application 
> >> master
> >> memory is set to 2048, and my -Xmx setting is 

[jira] [Updated] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event

2015-11-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6424:
---
Labels: yarn-2928-1st-milestone  (was: )

> Store MR counters as timeline metrics instead of event
> --
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Junping Du
>  Labels: yarn-2928-1st-milestone
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from 
> JobFinishedEvent as timeline events with counters details in JSON format. 
> We need to store framework specific counters as metrics in timeline service 
> to support query, aggregation, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (MAPREDUCE-3065) ApplicationMaster killed by NodeManager due to excessive virtual memory consumption

2015-11-13 Thread Vinod Kumar Vavilapalli (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved MAPREDUCE-3065.

Resolution: Duplicate

Resolving correctly as a dup of MAPREDUCE-3068.

> ApplicationMaster killed by NodeManager due to excessive virtual memory 
> consumption
> ---
>
> Key: MAPREDUCE-3065
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3065
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.0.0-alpha
>Reporter: Chris Riccomini
>
> > Hey Vinod,
> > 
> > OK, so I have a little more clarity into this.
> > 
> > When I bump my resource request for my AM to 4096, it runs. The important 
> > line in the NM logs is:
> > 
> > 2011-09-21 13:43:44,366 INFO  monitor.ContainersMonitorImpl 
> > (ContainersMonitorImpl.java:run(402)) - Memory usage of ProcessTree 25656 
> > for container-id container_1316637655278_0001_01_01 : Virtual 
> > 2260938752 bytes, limit : 4294967296 bytes; Physical 120860672 bytes, limit 
> > -1 bytes
> > 
> > The thing to note is the virtual memory, which is off the charts, even 
> > though my physical memory is almost nothing (12 megs). I'm still poking 
> > around the code, but I am noticing that there are two checks in the NM, one 
> > for virtual mem, and one for physical mem. The virtual memory check appears 
> > to be toggle-able, but is presumably defaulted to on.
> > 
> > At this point I'm trying to figure out exactly what the VMEM check is for, 
> > why YARN thinks my app is taking 2 gigs, and how to fix this.
> > 
> > Cheers,
> > Chris
> > 
> > From: Chris Riccomini [criccom...@linkedin.com]
> > Sent: Wednesday, September 21, 2011 1:42 PM
> > To: mapreduce-...@hadoop.apache.org
> > Subject: Re: ApplicationMaster Memory Usage
> > 
> > For the record, I bumped to 4096 for memory resource request, and it works.
> > :(
> > 
> > 
> > On 9/21/11 1:32 PM, "Chris Riccomini"  wrote:
> > 
> >> Hey Vinod,
> >> 
> >> So, I ran my application master directly from the CLI. I commented out the
> >> YARN-specific code. It runs fine without leaking memory.
> >> 
> >> I then ran it from YARN, with all YARN-specific code commented it. It again
> >> ran fine.
> >> 
> >> I then uncommented JUST my registerWithResourceManager call. It then fails
> >> with OOM after a few seconds. I call registerWithResourceManager, and then 
> >> go
> >> into a while(true) { println("yeh") sleep(1000) }. Doing this prints:
> >> 
> >> yeh
> >> yeh
> >> yeh
> >> yeh
> >> yeh
> >> 
> >> At which point, it dies, and, in the NodeManager,I see:
> >> 
> >> 2011-09-21 13:24:51,036 WARN  monitor.ContainersMonitorImpl
> >> (ContainersMonitorImpl.java:isProcessTreeOverLimit(289)) - Process tree for
> >> container: container_1316626117280_0005_01_01 has processes older than 
> >> 1
> >> iteration running over the configured limit. Limit=2147483648, current 
> >> usage =
> >> 2192773120
> >> 2011-09-21 13:24:51,037 WARN  monitor.ContainersMonitorImpl
> >> (ContainersMonitorImpl.java:run(453)) - Container
> >> [pid=23852,containerID=container_1316626117280_0005_01_01] is running
> >> beyond memory-limits. Current usage : 2192773120bytes. Limit :
> >> 2147483648bytes. Killing container.
> >> Dump of the process-tree for container_1316626117280_0005_01_01 :
> >> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> >> SYSTEM_TIME(MILLIS)
> >> VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> >> |- 23852 20570 23852 23852 (bash) 0 0 108638208 303 /bin/bash -c java 
> >> -Xmx512M
> >> -cp './package/*' kafka.yarn.ApplicationMaster
> >> /home/criccomi/git/kafka-yarn/dist/kafka-streamer.tgz 5 1 1316626117280
> >> com.linkedin.TODO 1
> >> 1>/tmp/logs/application_1316626117280_0005/container_1316626117280_0005_01_000
> >> 001/stdout
> >> 2>/tmp/logs/application_1316626117280_0005/container_1316626117280_0005_01_000
> >> 001/stderr
> >> |- 23855 23852 23852 23852 (java) 81 4 2084134912 14772 java -Xmx512M -cp
> >> ./package/* kafka.yarn.ApplicationMaster
> >> /home/criccomi/git/kafka-yarn/dist/kafka-streamer.tgz 5 1 1316626117280
> >> com.linkedin.TODO 1
> >> 2011-09-21 13:24:51,037 INFO  monitor.ContainersMonitorImpl
> >> (ContainersMonitorImpl.java:run(463)) - Removed ProcessTree with root 23852
> >> 
> >> Either something is leaking in YARN, or my registerWithResourceManager code
> >> (see below) is doing something funky.
> >> 
> >> I'm trying to avoid going through all the pain of attaching a remote 
> >> debugger.
> >> Presumably things aren't leaking in YARN, which means it's likely that I'm
> >> doing something wrong in my registration code.
> >> 
> >> Incidentally, my NodeManager is running with 1000 megs. My 

[jira] [Resolved] (MAPREDUCE-1901) Jobs should not submit the same jar files over and over again

2015-11-13 Thread Vinod Kumar Vavilapalli (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli resolved MAPREDUCE-1901.

Resolution: Duplicate

YARN-1492 implemented a solution reasonably close to what [~jsensarma] 
proposed. Closing this very old JIRA as a dup, please reopen if you disagree.

> Jobs should not submit the same jar files over and over again
> -
>
> Key: MAPREDUCE-1901
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1901
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Joydeep Sen Sarma
> Attachments: 1901.PATCH, 1901.PATCH
>
>
> Currently each Hadoop job uploads the required resources 
> (jars/files/archives) to a new location in HDFS. Map-reduce nodes involved in 
> executing this job would then download these resources into local disk.
> In an environment where most of the users are using a standard set of jars 
> and files (because they are using a framework like Hive/Pig) - the same jars 
> keep getting uploaded and downloaded repeatedly. The overhead of this 
> protocol (primarily in terms of end-user latency) is significant when:
> - the jobs are small (and conversantly - large in number)
> - Namenode is under load (meaning hdfs latencies are high and made worse, in 
> part, by this protocol)
> Hadoop should provide a way for jobs in a cooperative environment to not 
> submit the same files over and again. Identifying and caching execution 
> resources by a content signature (md5/sha) would be a good alternative to 
> have available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance

2015-11-13 Thread Sangjin Lee (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangjin Lee updated MAPREDUCE-6372:
---
Labels: yarn-2928-1st-milestone  (was: )

> clean up several issues with TimelineServicePerformance
> ---
>
> Key: MAPREDUCE-6372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372
> Project: Hadoop Map/Reduce
>  Issue Type: Sub-task
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>  Labels: yarn-2928-1st-milestone
>
> We found a few issues with the TimelineServicePerformanceV2 test driver while 
> running it for the performance tests. Filing this JIRA to fix those issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (MAPREDUCE-5889) Deprecate FileInputFormat.setInputPaths(Job, String) and FileInputFormat.addInputPaths(Job, String)

2015-11-13 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004619#comment-15004619
 ] 

Hadoop QA commented on MAPREDUCE-5889:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 6s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 8 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 26s 
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 14s 
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
25s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 25s 
{color} | {color:red} hadoop-tools/hadoop-datajoin in trunk has 2 extant 
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s 
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s 
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 25s 
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 16s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 27s 
{color} | {color:red} 
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_60
 with JDK v1.8.0_60 generated 1 new issues (was 100, now 100). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s 
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 33s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with 
JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 7s {color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK 
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s 
{color} | {color:green} hadoop-mapreduce-examples in the patch passed with JDK 
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s 
{color} | {color:green} hadoop-datajoin in the patch passed with JDK v1.8.0_60. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 9s 
{color} | {color:green} hadoop-gridmix in the patch passed with JDK v1.8.0_60. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 16s 

[jira] [Updated] (MAPREDUCE-5889) Deprecate FileInputFormat.setInputPaths(Job, String) and FileInputFormat.addInputPaths(Job, String)

2015-11-13 Thread Akira AJISAKA (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-5889:
-
Attachment: MAPREDUCE-5889.5.patch

v05: Fix javadoc warning.
Findbugs warnings are not related to the patch. This issue is tracked by 
HADOOP-12378.

> Deprecate FileInputFormat.setInputPaths(Job, String) and 
> FileInputFormat.addInputPaths(Job, String)
> ---
>
> Key: MAPREDUCE-5889
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5889
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
>  Labels: BB2015-05-TBR, newbie
> Attachments: MAPREDUCE-5889.3.patch, MAPREDUCE-5889.4.patch, 
> MAPREDUCE-5889.5.patch, MAPREDUCE-5889.patch, MAPREDUCE-5889.patch
>
>
> {{FileInputFormat.setInputPaths(Job job, String commaSeparatedPaths)}} and 
> {{FileInputFormat.addInputPaths(Job job, String commaSeparatedPaths)}} fail 
> to parse commaSeparatedPaths if a comma is included in the file path. (e.g. 
> Path: {{/path/file,with,comma}})
> We should deprecate these methods and document to use {{setInputPaths(Job 
> job, Path... inputPaths)}} and {{addInputPaths(Job job, Path... inputPaths)}} 
> instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page

2015-11-13 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004054#comment-15004054
 ] 

Hadoop QA commented on MAPREDUCE-6499:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 30s 
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 20s 
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s 
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 18s 
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 18s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 8s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 8s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 44s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 37s 
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK 
v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 14s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_79. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 59s 
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK 
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s 
{color} | {color:red} Patch generated 14 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 37s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests | hadoop.fs.TestLocalFsFCStatistics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1 

[jira] [Commented] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page

2015-11-13 Thread Akira AJISAKA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004057#comment-15004057
 ] 

Akira AJISAKA commented on MAPREDUCE-6499:
--

Tried your patch and it looks good to me, +1. Attaching a screen shot.

> Add More Informations Of Jobs In JobHistory Main Page
> -
>
> Key: MAPREDUCE-6499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Trivial
> Attachments: HADOOP-MAPREDUCE-6499.2.patch, 
> HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, 
> HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.6.patch, 
> HADOOP-MAPREDUCE-6499.7.patch, HADOOP-MAPREDUCE-6499.8.patch, 
> HADOOP-MAPREDUCE-6499.9.patch, HADOOP-MAPREDUCE-6499.patch
>
>
> Now in  JobHistory Main Page show too little information about finished 
> jobs,even don't have the job's running total time,only 
> startTime,finishedTime.So,in jobHistory main page,we can add more 
> informations about jobs, that we can better analyze jobs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (MAPREDUCE-6499) Add More Informations Of Jobs In JobHistory Main Page

2015-11-13 Thread Akira AJISAKA (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira AJISAKA updated MAPREDUCE-6499:
-
Attachment: Screen Shot 2015-11-13 at 23.47.10.png

> Add More Informations Of Jobs In JobHistory Main Page
> -
>
> Key: MAPREDUCE-6499
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6499
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
>Priority: Trivial
> Attachments: HADOOP-MAPREDUCE-6499.2.patch, 
> HADOOP-MAPREDUCE-6499.3.patch, HADOOP-MAPREDUCE-6499.4.patch, 
> HADOOP-MAPREDUCE-6499.5.patch, HADOOP-MAPREDUCE-6499.6.patch, 
> HADOOP-MAPREDUCE-6499.7.patch, HADOOP-MAPREDUCE-6499.8.patch, 
> HADOOP-MAPREDUCE-6499.9.patch, HADOOP-MAPREDUCE-6499.patch, Screen Shot 
> 2015-11-13 at 23.47.10.png
>
>
> Now in  JobHistory Main Page show too little information about finished 
> jobs,even don't have the job's running total time,only 
> startTime,finishedTime.So,in jobHistory main page,we can add more 
> informations about jobs, that we can better analyze jobs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe

2015-11-13 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15004003#comment-15004003
 ] 

Hadoop QA commented on MAPREDUCE-6542:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s 
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
4s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 24s 
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 11s 
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s 
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s 
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 29s 
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 12s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 12s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 56s 
{color} | {color:red} Patch generated 2 new checkstyle issues in root (total 
was 299, now 300). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 59s 
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_60. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 36s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with 
JDK v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 49s {color} 
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s 
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with 
JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s 
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 15s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests | hadoop.ipc.TestDecayRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| 

[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe

2015-11-13 Thread zhangyubiao (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003953#comment-15003953
 ] 

zhangyubiao commented on MAPREDUCE-6542:


[~djp] Thank you for your help. 
MAPREDUCE-6542-v3.patch for review

> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop  
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch, 
> MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before 
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
>  public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n", 
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
> 
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
> 
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
> 
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> But I find I query the SubmitTime and LaunchTime in hive and compare 
> JobHistory File time , I find that the submitTime  and launchTime was wrong.
> Finally,I chang to use the FastDateFormat to parse the time format and the 
> time become right 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe

2015-11-13 Thread zhangyubiao (JIRA)

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhangyubiao updated MAPREDUCE-6542:
---
Attachment: MAPREDUCE-6542-v3.patch

> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop  
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch, 
> MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before 
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
>  public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n", 
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
> 
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
> 
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
> 
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> But I find I query the SubmitTime and LaunchTime in hive and compare 
> JobHistory File time , I find that the submitTime  and launchTime was wrong.
> Finally,I chang to use the FastDateFormat to parse the time format and the 
> time become right 
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)