[jira] [Commented] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[
https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216338#comment-16216338
]
Hadoop QA commented on MAPREDUCE-6988:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
The patch generated 6 new + 20 unchanged - 1 fixed = 26 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 3s{color}
| {color:red} hadoop-mapreduce-client-hs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.hs.TestHistoryFileManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:ca8ddc6 |
| JIRA Issue | MAPREDUCE-6988 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893652/MAPREDUCE-6988.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6f91d3dc80be 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b90750c |
| Default Java | 1.8.0_131 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7201/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7201/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7201/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7201/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Let JHS support different file systems for intermediate_done and done
>
[jira] [Updated] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated MAPREDUCE-6988: Attachment: MAPREDUCE-6988.001.patch This should fix style and test issues > Let JHS support different file systems for intermediate_done and done > - > > Key: MAPREDUCE-6988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.4 >Reporter: Johan Gustavsson >Priority: Minor > Fix For: 2.7.5 > > Attachments: MAPREDUCE-6988.000.patch, MAPREDUCE-6988.001.patch > > > Currently JHS uses filecontext to move files from intermediate_done to done > folder. Since filecontext limits the use to 1 filesystem it makes it harder > to use s3 as a storage for jhist files. By moving this to filesystem > interface we can set hdfs for intermediate storage and s3 as long term > storage therefore reducing the number of puts to s3 and removing the need for > all M/R containers to carry a s3 sdk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6989) [Umbrella] Uploader tool for Distributed Cache Deploy
[ https://issues.apache.org/jira/browse/MAPREDUCE-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated MAPREDUCE-6989: -- Attachment: MAPREDUCE-6989 Mapreduce framework uploader tool.pdf Uploading design doc > [Umbrella] Uploader tool for Distributed Cache Deploy > - > > Key: MAPREDUCE-6989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: MAPREDUCE-6989 Mapreduce framework uploader tool.pdf > > > The proposal is to create a tool that collects all available jars in the > Hadoop classpath and adds them to a single tarball file. It then uploads the > resulting archive to an HDFS directory. This saves the cluster administrator > from having to set this up manually for Distributed Cache Deploy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6984) MR AM to clean up temporary files from previous attempt
[ https://issues.apache.org/jira/browse/MAPREDUCE-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215975#comment-16215975 ] Haibo Chen commented on MAPREDUCE-6984: --- Manually trigger the jenkins job. > MR AM to clean up temporary files from previous attempt > --- > > Key: MAPREDUCE-6984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6984 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 3.0.0-beta1 >Reporter: Gergo Repas >Assignee: Gergo Repas > Attachments: MAPREDUCE-6984.000.patch, MAPREDUCE-6984.001.patch > > > When the MR AM restarts, the > outputDir/_temporary/appAttemptNumber directory > remains on HDFS, even though this directory is not used during the next > attempt if the restart has been done without recovery. So if recovery is not > used for the AM restart, then the deletion of this directory can be done > earlier (at the start of the next attempt). The benefit is that more free > HDFS space is available for the next attempt. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6989) [Umbrella] Uploader tool for Distributed Cache Deploy
[ https://issues.apache.org/jira/browse/MAPREDUCE-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated MAPREDUCE-6989: -- Summary: [Umbrella] Uploader tool for Distributed Cache Deploy (was: Uploader tool for Distributed Cache Deploy) > [Umbrella] Uploader tool for Distributed Cache Deploy > - > > Key: MAPREDUCE-6989 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6989 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > > The proposal is to create a tool that collects all available jars in the > Hadoop classpath and adds them to a single tarball file. It then uploads the > resulting archive to an HDFS directory. This saves the cluster administrator > from having to set this up manually for Distributed Cache Deploy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6989) Uploader tool for Distributed Cache Deploy
Miklos Szegedi created MAPREDUCE-6989: - Summary: Uploader tool for Distributed Cache Deploy Key: MAPREDUCE-6989 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6989 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Miklos Szegedi Assignee: Miklos Szegedi The proposal is to create a tool that collects all available jars in the Hadoop classpath and adds them to a single tarball file. It then uploads the resulting archive to an HDFS directory. This saves the cluster administrator from having to set this up manually for Distributed Cache Deploy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[
https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214869#comment-16214869
]
Hadoop QA commented on MAPREDUCE-6988:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
The patch generated 5 new + 20 unchanged - 1 fixed = 25 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 35s{color}
| {color:red} hadoop-mapreduce-client-hs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.hs.TestHistoryFileManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:ca8ddc6 |
| JIRA Issue | MAPREDUCE-6988 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893495/MAPREDUCE-6988.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4f39f19dbc40 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 248d9b6 |
| Default Java | 1.8.0_131 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7199/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7199/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7199/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7199/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Let JHS support different file systems for intermediate_done and done
>
[jira] [Updated] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated MAPREDUCE-6988: Target Version/s: 2.8.1 (was: 2.9.0, 2.7.4, 2.8.1, 3.0.0) > Let JHS support different file systems for intermediate_done and done > - > > Key: MAPREDUCE-6988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.4 >Reporter: Johan Gustavsson >Priority: Minor > Fix For: 2.7.5 > > Attachments: MAPREDUCE-6988.000.patch > > > Currently JHS uses filecontext to move files from intermediate_done to done > folder. Since filecontext limits the use to 1 filesystem it makes it harder > to use s3 as a storage for jhist files. By moving this to filesystem > interface we can set hdfs for intermediate storage and s3 as long term > storage therefore reducing the number of puts to s3 and removing the need for > all M/R containers to carry a s3 sdk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated MAPREDUCE-6988: Fix Version/s: (was: 2.8.3) (was: 3.0.0) (was: 2.9.0) > Let JHS support different file systems for intermediate_done and done > - > > Key: MAPREDUCE-6988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.4 >Reporter: Johan Gustavsson >Priority: Minor > Fix For: 2.7.5 > > Attachments: MAPREDUCE-6988.000.patch > > > Currently JHS uses filecontext to move files from intermediate_done to done > folder. Since filecontext limits the use to 1 filesystem it makes it harder > to use s3 as a storage for jhist files. By moving this to filesystem > interface we can set hdfs for intermediate storage and s3 as long term > storage therefore reducing the number of puts to s3 and removing the need for > all M/R containers to carry a s3 sdk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated MAPREDUCE-6988: Fix Version/s: 2.8.3 3.0.0 2.7.5 2.9.0 Affects Version/s: 3.0.0 2.9.0 2.8.0 2.7.4 Target Version/s: 2.8.1, 2.7.4, 2.9.0, 3.0.0 Status: Patch Available (was: Open) > Let JHS support different file systems for intermediate_done and done > - > > Key: MAPREDUCE-6988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.4, 2.8.0, 2.9.0, 3.0.0 >Reporter: Johan Gustavsson >Priority: Minor > Fix For: 2.9.0, 2.7.5, 3.0.0, 2.8.3 > > Attachments: MAPREDUCE-6988.000.patch > > > Currently JHS uses filecontext to move files from intermediate_done to done > folder. Since filecontext limits the use to 1 filesystem it makes it harder > to use s3 as a storage for jhist files. By moving this to filesystem > interface we can set hdfs for intermediate storage and s3 as long term > storage therefore reducing the number of puts to s3 and removing the need for > all M/R containers to carry a s3 sdk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated MAPREDUCE-6988: Affects Version/s: (was: 3.0.0) (was: 2.9.0) (was: 2.8.0) > Let JHS support different file systems for intermediate_done and done > - > > Key: MAPREDUCE-6988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.4 >Reporter: Johan Gustavsson >Priority: Minor > Fix For: 2.7.5 > > Attachments: MAPREDUCE-6988.000.patch > > > Currently JHS uses filecontext to move files from intermediate_done to done > folder. Since filecontext limits the use to 1 filesystem it makes it harder > to use s3 as a storage for jhist files. By moving this to filesystem > interface we can set hdfs for intermediate storage and s3 as long term > storage therefore reducing the number of puts to s3 and removing the need for > all M/R containers to carry a s3 sdk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
[ https://issues.apache.org/jira/browse/MAPREDUCE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated MAPREDUCE-6988: Attachment: MAPREDUCE-6988.000.patch > Let JHS support different file systems for intermediate_done and done > - > > Key: MAPREDUCE-6988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Johan Gustavsson >Priority: Minor > Attachments: MAPREDUCE-6988.000.patch > > > Currently JHS uses filecontext to move files from intermediate_done to done > folder. Since filecontext limits the use to 1 filesystem it makes it harder > to use s3 as a storage for jhist files. By moving this to filesystem > interface we can set hdfs for intermediate storage and s3 as long term > storage therefore reducing the number of puts to s3 and removing the need for > all M/R containers to carry a s3 sdk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6988) Let JHS support different file systems for intermediate_done and done
Johan Gustavsson created MAPREDUCE-6988: --- Summary: Let JHS support different file systems for intermediate_done and done Key: MAPREDUCE-6988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6988 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Johan Gustavsson Priority: Minor Currently JHS uses filecontext to move files from intermediate_done to done folder. Since filecontext limits the use to 1 filesystem it makes it harder to use s3 as a storage for jhist files. By moving this to filesystem interface we can set hdfs for intermediate storage and s3 as long term storage therefore reducing the number of puts to s3 and removing the need for all M/R containers to carry a s3 sdk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6987) JHS Log Scanner and Cleaner blocked
[
https://issues.apache.org/jira/browse/MAPREDUCE-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated MAPREDUCE-6987:
Affects Version/s: 2.9.0
Target Version/s: 2.9.0, 3.0.0, 3.1.0
> JHS Log Scanner and Cleaner blocked
> ---
>
> Key: MAPREDUCE-6987
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6987
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.9.0, 3.0.0-alpha1
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
>
> {code}
> "Log Scanner/Cleaner #1" #81 prio=5 os_prio=0 tid=0x7fd6c010f000
> nid=0x11db waiting on condition [0x7fd6aa859000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd6c88a80> (a
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
> at
> org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "Log Scanner/Cleaner #0" #80 prio=5 os_prio=0 tid=0x7fd6c010c800
> nid=0x11da waiting on condition [0x7fd6aa95a000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xd6c8> (a
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
> at
> org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Both threads waiting on {{FutureTask.get()}} for infinite time after first
> execution
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
