[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135102#comment-15135102
]
Kai Sasaki commented on MAPREDUCE-6607:
---
Could someone review this?
> .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or
> mapreduce.task.files.preserve.filepattern are set
> ---
>
> Key: MAPREDUCE-6607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Maysam Yabandeh
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6607.01.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned
> up:
> * mapreduce.task.files.preserve.failedtask
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter
> was supposed to be used only if that task name matches against the specified
> regular expression.
> {code}
> protected boolean keepJobFiles(JobConf conf) {
> return (conf.getKeepTaskFilesPattern() != null || conf
> .getKeepFailedTaskFiles());
> }
> {code}
> {code}
> public void cleanupStagingDir() throws IOException {
> /* make sure we clean the staging files */
> String jobTempDir = null;
> FileSystem fs = getFileSystem(getConfig());
> try {
> if (!keepJobFiles(new JobConf(getConfig( {
> jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
> if (jobTempDir == null) {
> LOG.warn("Job Staging directory is null");
> return;
> }
> Path jobTempDirPath = new Path(jobTempDir);
> LOG.info("Deleting staging directory " +
> FileSystem.getDefaultUri(getConfig()) +
> " " + jobTempDir);
> fs.delete(jobTempDirPath, true);
> }
> } catch(IOException io) {
> LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6110) JobHistoryServer CLI throws NullPointerException with job ids that do not exist
[
https://issues.apache.org/jira/browse/MAPREDUCE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Sasaki updated MAPREDUCE-6110:
--
Attachment: MAPREDUCE-6110.02.patch
> JobHistoryServer CLI throws NullPointerException with job ids that do not
> exist
> ---
>
> Key: MAPREDUCE-6110
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6110
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: Li Lu
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6110.01.patch, MAPREDUCE-6110.02.patch
>
>
> When using JobHistoryServer CLI to query a job id that does not exist on the
> server, it may throw NullPointerException sometimes.
> I tried "mapred job -events 0 100", and the result was:
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.tools.CLI.listEvents(CLI.java:487)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1237)
> Similar symptoms also appear with -list-attempt-ids, but were fine with
> -status and -set-priority.
> I traced back to CLI.listEvents, and line 487 is:
> {code}
> TaskCompletionEvent[] events = job.
> getTaskCompletionEvents(fromEventId, numEvents);
> {code}
> The job object is obtained from JobID.forName(jobid)) (line 316), which will
> return null if the job does not exist on server.
> Maybe we want to have some behaviors consistent with -status here, by simply
> reporting jobId does not exist?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6424) Store MR counters as timeline metrics instead of event
[ https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134785#comment-15134785 ] Naganarasimha G R commented on MAPREDUCE-6424: -- Hi [~djp], I would like to work on this issue if you are not planning to work on this. > Store MR counters as timeline metrics instead of event > -- > > Key: MAPREDUCE-6424 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: yarn-2928-1st-milestone > > In MAPREDUCE-6327, we make map/reduce counters get encoded from > JobFinishedEvent as timeline events with counters details in JSON format. > We need to store framework specific counters as metrics in timeline service > to support query, aggregation, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6110) JobHistoryServer CLI throws NullPointerException with job ids that do not exist
[
https://issues.apache.org/jira/browse/MAPREDUCE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135278#comment-15135278
]
Hadoop QA commented on MAPREDUCE-6110:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
patch generated 1 new + 50 unchanged - 1 fixed = 51 total (was 51) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 23s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786575/MAPREDUCE-6110.02.patch
|
| JIRA Issue | MAPREDUCE-6110 |
| Optional
[jira] [Commented] (MAPREDUCE-6579) Test failure : TestNetworkedJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135570#comment-15135570
]
Naganarasimha G R commented on MAPREDUCE-6579:
--
Seems like this failure is shown frequently in other jira's, [~ajisakaa] can
you get it committed as [~wangda] has agreed as +1, else [~rohithsharma] can
you get this committed ?
> Test failure : TestNetworkedJob
> ---
>
> Key: MAPREDUCE-6579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: MAPREDUCE-6579.01.patch, MAPREDUCE-6579.02.patch,
> MAPREDUCE-6579.03.patch, MAPREDUCE-6579.04.patch
>
>
> From
> [https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
> {code}
> Running org.apache.hadoop.mapred.TestNetworkedJob
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
> testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
> 30.55 sec <<< FAILURE!
> org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
> Application is Activated, waiting for resources to be assigned for AM.
> Details : AM Partition = ; Partition Resource =
> ; Queue's Absolute capacity = 100.0 % ; Queue's
> Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]>
> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
