[jira] [Commented] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057029#comment-16057029 ] Kai Zheng commented on HDFS-11943: -- Thanks [~liaoyuxiangqin] for the update. 1. Please also check and do the same thing for AbstractNativeRawDecoder. When you ensure the base class to indicate the preference, the method can be removed from sub classes. 2. Note a minor check style issue. > Warn log frequently print to screen in doEncode function on > AbstractNativeRawEncoder class > --- > > Key: HDFS-11943 > URL: https://issues.apache.org/jira/browse/HDFS-11943 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, native >Affects Versions: 3.0.0-alpha4 > Environment: cluster: 3 nodes > os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, > Ubuntu4.4.0-31-generic) > hadoop version: hadoop-3.0.0-alpha4 > erasure coding: XOR-2-1-64k and enabled Intel ISA-L > hadoop fs -put file / >Reporter: liaoyuxiangqin >Assignee: liaoyuxiangqin >Priority: Minor > Attachments: HDFS-11943.002.patch, HDFS-11943.003.patch, > HDFS-11943.patch > > Original Estimate: 0.05h > Remaining Estimate: 0.05h > > when i write file to hdfs on above environment, the hdfs client frequently > print warn log of use direct ByteBuffer inputs/outputs in doEncode function > to screen, detail information as follows: > 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: > convertToByteBufferState is invoked, not efficiently. Please use direct > ByteBuffer inputs/outputs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11933) Arguments check for ErasureCodingPolicy->composePolicyName
[ https://issues.apache.org/jira/browse/HDFS-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-11933: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha4 Target Version/s: 3.0.0-alpha4 (was: 2.9.0, 3.0.0-alpha4) Status: Resolved (was: Patch Available) Committed to trunk. Thanks [~figo] for the contribution and [~vagarychen] for the review! > Arguments check for ErasureCodingPolicy->composePolicyName > -- > > Key: HDFS-11933 > URL: https://issues.apache.org/jira/browse/HDFS-11933 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei >Priority: Minor > Fix For: 3.0.0-alpha4 > > Attachments: HDFS-11933.001.patch > > > Function composePolicyName is called by ErasureCodingPolicy, but both of them > are not judge the arguments not NULL.It 's better to judge in the function > composePolicyName. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11989) Ozone: add TestKeysRatis, TestBucketsRatis and TestVolumeRatis
[ https://issues.apache.org/jira/browse/HDFS-11989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11989: --- Summary: Ozone: add TestKeysRatis, TestBucketsRatis and TestVolumeRatis (was: Ozone: add TestKeysRatis and TestBucketsRatis) Description: Add Ratis tests similar to TestKeys, TestBuckets and TestVolume. (was: Add Ratis tests similar to TestKeys and TestBuckets.) > Ozone: add TestKeysRatis, TestBucketsRatis and TestVolumeRatis > -- > > Key: HDFS-11989 > URL: https://issues.apache.org/jira/browse/HDFS-11989 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11989-HDFS-7240.20170618.patch, > HDFS-11989-HDFS-7240.20170620b.patch, HDFS-11989-HDFS-7240.20170620c.patch, > HDFS-11989-HDFS-7240.20170620.patch, HDFS-11989-HDFS-7240.20170621.patch > > > Add Ratis tests similar to TestKeys, TestBuckets and TestVolume. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11989) Ozone: add TestKeysRatis and TestBucketsRatis
[ https://issues.apache.org/jira/browse/HDFS-11989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11989: --- Attachment: HDFS-11989-HDFS-7240.20170621.patch HDFS-11989-HDFS-7240.20170621.patch: adds also TestVolumeRatis. > Ozone: add TestKeysRatis and TestBucketsRatis > - > > Key: HDFS-11989 > URL: https://issues.apache.org/jira/browse/HDFS-11989 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11989-HDFS-7240.20170618.patch, > HDFS-11989-HDFS-7240.20170620b.patch, HDFS-11989-HDFS-7240.20170620c.patch, > HDFS-11989-HDFS-7240.20170620.patch, HDFS-11989-HDFS-7240.20170621.patch > > > Add Ratis tests similar to TestKeys and TestBuckets. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11933) Arguments check for ErasureCodingPolicy->composePolicyName
[ https://issues.apache.org/jira/browse/HDFS-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-11933: - Summary: Arguments check for ErasureCodingPolicy->composePolicyName (was: The function composePolicyName should judge arguments not NULL) > Arguments check for ErasureCodingPolicy->composePolicyName > -- > > Key: HDFS-11933 > URL: https://issues.apache.org/jira/browse/HDFS-11933 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei >Priority: Minor > Attachments: HDFS-11933.001.patch > > > Function composePolicyName is called by ErasureCodingPolicy, but both of them > are not judge the arguments not NULL.It 's better to judge in the function > composePolicyName. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11933) The function composePolicyName should judge arguments not NULL
[ https://issues.apache.org/jira/browse/HDFS-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057020#comment-16057020 ] Kai Zheng commented on HDFS-11933: -- It's good to have the check and added tests. +1. > The function composePolicyName should judge arguments not NULL > --- > > Key: HDFS-11933 > URL: https://issues.apache.org/jira/browse/HDFS-11933 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei >Priority: Minor > Attachments: HDFS-11933.001.patch > > > Function composePolicyName is called by ErasureCodingPolicy, but both of them > are not judge the arguments not NULL.It 's better to judge in the function > composePolicyName. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11933) The function composePolicyName should judge arguments not NULL
[ https://issues.apache.org/jira/browse/HDFS-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng reassigned HDFS-11933: Assignee: lufei > The function composePolicyName should judge arguments not NULL > --- > > Key: HDFS-11933 > URL: https://issues.apache.org/jira/browse/HDFS-11933 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei >Priority: Minor > Attachments: HDFS-11933.001.patch > > > Function composePolicyName is called by ErasureCodingPolicy, but both of them > are not judge the arguments not NULL.It 's better to judge in the function > composePolicyName. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11493) Ozone: SCM: Add the ability to handle container reports
[
https://issues.apache.org/jira/browse/HDFS-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057015#comment-16057015
]
Hadoop QA commented on HDFS-11493:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
33s{color} | {color:red} hadoop-common-project/hadoop-common in HDFS-7240 has
19 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 52s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}236m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.scm.node.TestNodeManager |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11493 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873773/HDFS-11493-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 1e29c79decb1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64
[jira] [Commented] (HDFS-11992) Replace commons-logging APIs with slf4j in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056971#comment-16056971
]
hu xiaodong commented on HDFS-11992:
thanks [~ajisakaa], I will create a patch for branch-2 after HADOOP-14542 is
fixed.
> Replace commons-logging APIs with slf4j in FsDatasetImpl
>
>
> Key: HDFS-11992
> URL: https://issues.apache.org/jira/browse/HDFS-11992
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11992.001.patch, HDFS-11992.002.patch
>
>
> {{FsDatasetImpl.LOG}} is widely used and this will change the APIs of
> InstrumentedLock and InstrumentedWriteLock, so this issue is to change only
> {{FsDatasetImpl.LOG}} and other related APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[
https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056951#comment-16056951
]
Hadoop QA commented on HDFS-11943:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
59s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 17
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11943 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873778/HDFS-11943.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c6921aab5cda 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1a59847 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19975/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19975/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19975/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19975/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Warn log frequently print to screen in doEncode function on
> AbstractNativeRawEncoder class
> ---
>
> Key: HDFS-11943
> URL: https://issues.apache.org/jira/browse/HDFS-11943
>
[jira] [Commented] (HDFS-11992) Replace commons-logging APIs with slf4j in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056900#comment-16056900
]
Hudson commented on HDFS-11992:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11899 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11899/])
HDFS-11992. Replace commons-logging APIs with slf4j in FsDatasetImpl.
(aajisaka: rev 1a598479a9faec787706bcf924dfbd88a80e1b82)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/InstrumentedReadWriteLock.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/InstrumentedReadLock.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestInstrumentedLock.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/InstrumentedLock.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestInstrumentedReadWriteLock.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/InstrumentedWriteLock.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
> Replace commons-logging APIs with slf4j in FsDatasetImpl
>
>
> Key: HDFS-11992
> URL: https://issues.apache.org/jira/browse/HDFS-11992
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11992.001.patch, HDFS-11992.002.patch
>
>
> {{FsDatasetImpl.LOG}} is widely used and this will change the APIs of
> InstrumentedLock and InstrumentedWriteLock, so this issue is to change only
> {{FsDatasetImpl.LOG}} and other related APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11983) Ozone: Add documentation for metrics in KSMMetrics to OzoneMetrics.md
[ https://issues.apache.org/jira/browse/HDFS-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11983: - Summary: Ozone: Add documentation for metrics in KSMMetrics to OzoneMetrics.md (was: Add documentation for metrics in KSMMetrics to OzoneMetrics.md) > Ozone: Add documentation for metrics in KSMMetrics to OzoneMetrics.md > - > > Key: HDFS-11983 > URL: https://issues.apache.org/jira/browse/HDFS-11983 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation, ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Fix For: HDFS-7240 > > Attachments: HDFS-11983-HDFS-7240.001.patch, > HDFS-11983-HDFS-7240.002.patch > > > Metrics defined in KSMMetrics are not documented in OzoneMetrics.md. This > JIRA will track on this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11983) Add documentation for metrics in KSMMetrics to OzoneMetrics.md
[ https://issues.apache.org/jira/browse/HDFS-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11983: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) Committed this to feature branch. Thanks [~anu]! > Add documentation for metrics in KSMMetrics to OzoneMetrics.md > -- > > Key: HDFS-11983 > URL: https://issues.apache.org/jira/browse/HDFS-11983 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation, ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Fix For: HDFS-7240 > > Attachments: HDFS-11983-HDFS-7240.001.patch, > HDFS-11983-HDFS-7240.002.patch > > > Metrics defined in KSMMetrics are not documented in OzoneMetrics.md. This > JIRA will track on this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11963) Ozone: Documentation: Add getting started page
[ https://issues.apache.org/jira/browse/HDFS-11963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11963: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~anu] for the review. Committed addendum patch to feature branch. > Ozone: Documentation: Add getting started page > -- > > Key: HDFS-11963 > URL: https://issues.apache.org/jira/browse/HDFS-11963 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-11963-HDFS-7240.001.patch, > HDFS-11963-HDFS-7240.002.patch, HDFS-11963-HDFS-7240.003.patch, > HDFS-11963-HDFS-7240.004.patch, HDFS-11963-HDFS-7240.005.patch, > HDFS-11963-HDFS-7240.006.patch, HDFS-11963-HDFS-7240.addendum001.patch, > Screen Shot 2017-06-11 at 12.11.06 AM.png, Screen Shot 2017-06-11 at 12.11.19 > AM.png, Screen Shot 2017-06-11 at 12.11.32 AM.png > > > We need to add the Ozone section to hadoop documentation and also a section > on how to get started, that is what are configuration settings needed to > start running ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11963) Ozone: Documentation: Add getting started page
[ https://issues.apache.org/jira/browse/HDFS-11963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11963: - Fix Version/s: HDFS-7240 > Ozone: Documentation: Add getting started page > -- > > Key: HDFS-11963 > URL: https://issues.apache.org/jira/browse/HDFS-11963 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11963-HDFS-7240.001.patch, > HDFS-11963-HDFS-7240.002.patch, HDFS-11963-HDFS-7240.003.patch, > HDFS-11963-HDFS-7240.004.patch, HDFS-11963-HDFS-7240.005.patch, > HDFS-11963-HDFS-7240.006.patch, HDFS-11963-HDFS-7240.addendum001.patch, > Screen Shot 2017-06-11 at 12.11.06 AM.png, Screen Shot 2017-06-11 at 12.11.19 > AM.png, Screen Shot 2017-06-11 at 12.11.32 AM.png > > > We need to add the Ozone section to hadoop documentation and also a section > on how to get started, that is what are configuration settings needed to > start running ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11992) Replace commons-logging APIs with slf4j in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-11992:
-
Target Version/s: 2.9.0, 3.0.0-alpha4
Fix Version/s: 3.0.0-alpha4
Thanks! Committed this to trunk.
I noticed HADOOP-14542 is required when backporting to branch-2 due to the use
of {{IOUtils.cleanup(Logger, Closeable...)}}. Would you create a patch for
branch-2 after HADOOP-14542 is fixed?
> Replace commons-logging APIs with slf4j in FsDatasetImpl
>
>
> Key: HDFS-11992
> URL: https://issues.apache.org/jira/browse/HDFS-11992
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Fix For: 3.0.0-alpha4
>
> Attachments: HDFS-11992.001.patch, HDFS-11992.002.patch
>
>
> {{FsDatasetImpl.LOG}} is widely used and this will change the APIs of
> InstrumentedLock and InstrumentedWriteLock, so this issue is to change only
> {{FsDatasetImpl.LOG}} and other related APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11933) The function composePolicyName should judge arguments not NULL
[ https://issues.apache.org/jira/browse/HDFS-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056862#comment-16056862 ] lufei commented on HDFS-11933: -- Hi [~drankye] ，Can you do me a faver to review this code. Thanks. > The function composePolicyName should judge arguments not NULL > --- > > Key: HDFS-11933 > URL: https://issues.apache.org/jira/browse/HDFS-11933 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Priority: Minor > Attachments: HDFS-11933.001.patch > > > Function composePolicyName is called by ErasureCodingPolicy, but both of them > are not judge the arguments not NULL.It 's better to judge in the function > composePolicyName. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11992) Replace commons-logging APIs with slf4j in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056855#comment-16056855
]
Akira Ajisaka commented on HDFS-11992:
--
+1, checking this in.
> Replace commons-logging APIs with slf4j in FsDatasetImpl
>
>
> Key: HDFS-11992
> URL: https://issues.apache.org/jira/browse/HDFS-11992
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha3
>Reporter: Akira Ajisaka
>Assignee: hu xiaodong
> Attachments: HDFS-11992.001.patch, HDFS-11992.002.patch
>
>
> {{FsDatasetImpl.LOG}} is widely used and this will change the APIs of
> InstrumentedLock and InstrumentedWriteLock, so this issue is to change only
> {{FsDatasetImpl.LOG}} and other related APIs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11998) Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056845#comment-16056845
]
Yiqun Lin commented on HDFS-11998:
--
Hi [~vagarychen], thanks for filing this. In the next patch, please don't
forget updating the default value in class {{DFSConfigKeys.java}}.
{code}
public static final String DFS_USE_DFS_NETWORK_TOPOLOGY_KEY =
"dfs.use.dfs.network.topology";
public static final boolean DFS_USE_DFS_NETWORK_TOPOLOGY_DEFAULT = false;
{code}
> Enable DFSNetworkTopology as default
>
>
> Key: HDFS-11998
> URL: https://issues.apache.org/jira/browse/HDFS-11998
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11998.001.patch
>
>
> HDFS-11530 has made it configurable to use {{DFSNetworkTopology}}, and still
> uses {{NetworkTopology}} as default.
> Given the stress testing in HDFS-11923 which shows the correctness of
> DFSNetworkTopology, and the performance testing in HDFS-11535 which shows how
> DFSNetworkTopology can outperform NetworkTopology. I think we are at the
> point where I can and should enable DFSNetworkTopology as default.
> Any comments/thoughts are more than welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-11943: -- Status: Patch Available (was: Open) > Warn log frequently print to screen in doEncode function on > AbstractNativeRawEncoder class > --- > > Key: HDFS-11943 > URL: https://issues.apache.org/jira/browse/HDFS-11943 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, native >Affects Versions: 3.0.0-alpha4 > Environment: cluster: 3 nodes > os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, > Ubuntu4.4.0-31-generic) > hadoop version: hadoop-3.0.0-alpha4 > erasure coding: XOR-2-1-64k and enabled Intel ISA-L > hadoop fs -put file / >Reporter: liaoyuxiangqin >Assignee: liaoyuxiangqin >Priority: Minor > Attachments: HDFS-11943.002.patch, HDFS-11943.003.patch, > HDFS-11943.patch > > Original Estimate: 0.05h > Remaining Estimate: 0.05h > > when i write file to hdfs on above environment, the hdfs client frequently > print warn log of use direct ByteBuffer inputs/outputs in doEncode function > to screen, detail information as follows: > 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: > convertToByteBufferState is invoked, not efficiently. Please use direct > ByteBuffer inputs/outputs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-11943: -- Attachment: HDFS-11943.003.patch > Warn log frequently print to screen in doEncode function on > AbstractNativeRawEncoder class > --- > > Key: HDFS-11943 > URL: https://issues.apache.org/jira/browse/HDFS-11943 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, native >Affects Versions: 3.0.0-alpha4 > Environment: cluster: 3 nodes > os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, > Ubuntu4.4.0-31-generic) > hadoop version: hadoop-3.0.0-alpha4 > erasure coding: XOR-2-1-64k and enabled Intel ISA-L > hadoop fs -put file / >Reporter: liaoyuxiangqin >Assignee: liaoyuxiangqin >Priority: Minor > Attachments: HDFS-11943.002.patch, HDFS-11943.003.patch, > HDFS-11943.patch > > Original Estimate: 0.05h > Remaining Estimate: 0.05h > > when i write file to hdfs on above environment, the hdfs client frequently > print warn log of use direct ByteBuffer inputs/outputs in doEncode function > to screen, detail information as follows: > 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: > convertToByteBufferState is invoked, not efficiently. Please use direct > ByteBuffer inputs/outputs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-11943: -- Status: Open (was: Patch Available) > Warn log frequently print to screen in doEncode function on > AbstractNativeRawEncoder class > --- > > Key: HDFS-11943 > URL: https://issues.apache.org/jira/browse/HDFS-11943 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, native >Affects Versions: 3.0.0-alpha4 > Environment: cluster: 3 nodes > os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, > Ubuntu4.4.0-31-generic) > hadoop version: hadoop-3.0.0-alpha4 > erasure coding: XOR-2-1-64k and enabled Intel ISA-L > hadoop fs -put file / >Reporter: liaoyuxiangqin >Assignee: liaoyuxiangqin >Priority: Minor > Attachments: HDFS-11943.002.patch, HDFS-11943.patch > > Original Estimate: 0.05h > Remaining Estimate: 0.05h > > when i write file to hdfs on above environment, the hdfs client frequently > print warn log of use direct ByteBuffer inputs/outputs in doEncode function > to screen, detail information as follows: > 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: > convertToByteBufferState is invoked, not efficiently. Please use direct > ByteBuffer inputs/outputs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-11943: -- Attachment: (was: HDFS-11943.003.patch) > Warn log frequently print to screen in doEncode function on > AbstractNativeRawEncoder class > --- > > Key: HDFS-11943 > URL: https://issues.apache.org/jira/browse/HDFS-11943 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, native >Affects Versions: 3.0.0-alpha4 > Environment: cluster: 3 nodes > os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, > Ubuntu4.4.0-31-generic) > hadoop version: hadoop-3.0.0-alpha4 > erasure coding: XOR-2-1-64k and enabled Intel ISA-L > hadoop fs -put file / >Reporter: liaoyuxiangqin >Assignee: liaoyuxiangqin >Priority: Minor > Attachments: HDFS-11943.002.patch, HDFS-11943.patch > > Original Estimate: 0.05h > Remaining Estimate: 0.05h > > when i write file to hdfs on above environment, the hdfs client frequently > print warn log of use direct ByteBuffer inputs/outputs in doEncode function > to screen, detail information as follows: > 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: > convertToByteBufferState is invoked, not efficiently. Please use direct > ByteBuffer inputs/outputs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11493) Ozone: SCM: Add the ability to handle container reports
[ https://issues.apache.org/jira/browse/HDFS-11493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11493: Attachment: HDFS-11493-HDFS-7240.004.patch updated the patch to make sure all test failures are addressed. [~xyao] Can you please take a look. > Ozone: SCM: Add the ability to handle container reports > - > > Key: HDFS-11493 > URL: https://issues.apache.org/jira/browse/HDFS-11493 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: container-replication-storage.pdf, > exploring-scalability-scm.pdf, HDFS-11493-HDFS-7240.001.patch, > HDFS-11493-HDFS-7240.002.patch, HDFS-11493-HDFS-7240.003.patch, > HDFS-11493-HDFS-7240.004.patch > > > Once a datanode sends the container report it is SCM's responsibility to > determine if the replication levels are acceptable. If it is not, SCM should > initiate a replication request to another datanode. This JIRA tracks how SCM > handles a container report. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056748#comment-16056748
]
Hadoop QA commented on HDFS-12002:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
4s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestLocalDFS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12002 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873755/HDFS-12002-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 45a0e8f7bf6e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 185e3ad |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19973/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19973/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19973/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
>
[jira] [Commented] (HDFS-11125) [SPS]: Use smaller batches of BlockMovingInfo into the block storage movement command
[
https://issues.apache.org/jira/browse/HDFS-11125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056712#comment-16056712
]
Ewan Higgs commented on HDFS-11125:
---
The Provided Storage system is looking to use the SPS for asynchronously
writing files to a remote system ({{hdfs://}}, {{s3a://}}, {{wasb://}}). The
way the SPS currently works is file by file which suits the Provided Storage
system very well since it works on a file by file basis as well. Moving to use
smaller batches unlinked to the actual files would break current plans
discussed offline with [~umamaheswararao].
This could be worked through by offering the interface as
{{BlockStorageMovementCommandSchedulerSpi}} (or something less of a mouthful)
and implementing it one way for normal SPS and another way for Provided Storage.
> [SPS]: Use smaller batches of BlockMovingInfo into the block storage movement
> command
> -
>
> Key: HDFS-11125
> URL: https://issues.apache.org/jira/browse/HDFS-11125
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
>
> This is a follow-up task of HDFS-11068, where it sends all the blocks under a
> trackID over single heartbeat response(DNA_BLOCK_STORAGE_MOVEMENT command).
> If blocks are many under a given trackID(For example: a file contains many
> blocks) then those requests go across a network and come with a lot of
> overhead. In this jira, we will discuss and implement a mechanism to limit
> the list of items into smaller batches with in trackID.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11998) Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056705#comment-16056705
]
Chen Liang commented on HDFS-11998:
---
{{TestAvailableSpaceBlockPlacementPolicy}} fails because
{{AvailableSpaceBlockPlacementPolicy}} only rewrites chooseDataNode(...) for
it's optimization purpose. However the caller of this function in
{{BlockPlacementPolicyDefault#chooseRandom()}} has changed to call
chooseDataNode(…, type) on {{DFSNetworkTopology}}, this bypasses the change
that AvailableSpaceBlockPlacementPolicy has made in chooseDataNode(...), thus
invalidated its change. Need to also overwrite chooseDataNode(..., type) in
AvailableSpaceBlockPlacementPolicy. I've made the change locally which seems to
have solved the test failure. Will post next patch after the other test failure
gets resolved.
> Enable DFSNetworkTopology as default
>
>
> Key: HDFS-11998
> URL: https://issues.apache.org/jira/browse/HDFS-11998
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11998.001.patch
>
>
> HDFS-11530 has made it configurable to use {{DFSNetworkTopology}}, and still
> uses {{NetworkTopology}} as default.
> Given the stress testing in HDFS-11923 which shows the correctness of
> DFSNetworkTopology, and the performance testing in HDFS-11535 which shows how
> DFSNetworkTopology can outperform NetworkTopology. I think we are at the
> point where I can and should enable DFSNetworkTopology as default.
> Any comments/thoughts are more than welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11999) Ozone: Clarify startup error message of Datanode in case namenode is missing
[
https://issues.apache.org/jira/browse/HDFS-11999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11999:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-7240
Target Version/s: HDFS-7240
Status: Resolved (was: Patch Available)
[~elek] Thank you for the contribution. I have committed this to the feature
branch.
> Ozone: Clarify startup error message of Datanode in case namenode is missing
>
>
> Key: HDFS-11999
> URL: https://issues.apache.org/jira/browse/HDFS-11999
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Fix For: HDFS-7240
>
> Attachments: HDFS-11999-HDFS-7240.001.patch, HDFS-11999.patch
>
>
> Datanode is failing if namenode config setting is missing even for Ozone with
> a confusing error message:
> {code}
> 14:33:29.176 [main] ERROR o.a.h.hdfs.server.datanode.DataNode - Exception in
> secureMain
> java.io.IOException: No services to connect (NameNodes or SCM).
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.refreshNamenodes(BlockPoolManager.java:168)
> ~[classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1440)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:510)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2802)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2705)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2752)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2896)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2920)
> [classes/:na]
> 14:33:29.177 [main] INFO org.apache.hadoop.util.ExitUtil - Exiting with
> status 1: java.io.IOException: No services to connect (NameNodes or SCM).
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11988) Verify HDFS Snapshots with open files captured are safe across truncates and appends on current version file
[ https://issues.apache.org/jira/browse/HDFS-11988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056679#comment-16056679 ] Manoj Govindassamy commented on HDFS-11988: --- Above unit test failures are not related to the patch. > Verify HDFS Snapshots with open files captured are safe across truncates and > appends on current version file > > > Key: HDFS-11988 > URL: https://issues.apache.org/jira/browse/HDFS-11988 > Project: Hadoop HDFS > Issue Type: Test > Components: hdfs, snapshots >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Attachments: HDFS-11988.01.patch > > > With HDFS-11402 fix, open files can also be captured in the snapshots as > point-in-time copies. This jira is for adding more unit tests to verify the > correctness of the snapshot files using checksums when the current version of > the file is truncated and appended after the snapshot time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11998) Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056668#comment-16056668
]
Chen Liang commented on HDFS-11998:
---
{{TestAvailableSpaceBlockPlacementPolicy}} fail seems related,
{{TestReplicationPolicyWithNodeGroup}} also fails because node group tries to
cast new class to old ones and failed, still investigating these two failures.
The other fails all passed in my local run so should be unrelated.
> Enable DFSNetworkTopology as default
>
>
> Key: HDFS-11998
> URL: https://issues.apache.org/jira/browse/HDFS-11998
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11998.001.patch
>
>
> HDFS-11530 has made it configurable to use {{DFSNetworkTopology}}, and still
> uses {{NetworkTopology}} as default.
> Given the stress testing in HDFS-11923 which shows the correctness of
> DFSNetworkTopology, and the performance testing in HDFS-11535 which shows how
> DFSNetworkTopology can outperform NetworkTopology. I think we are at the
> point where I can and should enable DFSNetworkTopology as default.
> Any comments/thoughts are more than welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12005) Ozone: Web interface for SCM
[ https://issues.apache.org/jira/browse/HDFS-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056660#comment-16056660 ] Anu Engineer commented on HDFS-12005: - [~elek] Thanks for filing this JIRA. I am fine with any approach you choose. As you suggested let us stick to the jmx endpoint for now. It works well for namenode UI. I am fine with any framework that you would choose. One thing is that we might have to design the UI and then make sure all the needed counters are present in the SCM/KSM. > Ozone: Web interface for SCM > > > Key: HDFS-12005 > URL: https://issues.apache.org/jira/browse/HDFS-12005 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > > This is a propsal about how a web interface could be implemented for SCM (and > later for KSM) similar to the namenode ui. > 1. JS framework > There are three big option here. > A.) One is to use a full featured web framework with all the webpack/npm > minify/uglify magic. Build time the webpack/npm scripts should be run and the > result will be added to the jar file. > B.) It could be simplified if the generated minified/uglified js files are > added to the project on commit time. It requires an additional step for every > new patch (to generate the new minified javascripts) but doesn't require > additional JS build tools during the build. > C.) The third option is to make it as simple as possible similar to the > current namenode ui which uses javascript but every dependency is commited > (without JS minify/uglify and other preprocessing). > I prefer to the third one as: > * I have seen a lot of problems during frequent builds od older tez-ui > versions (bower version mismatch, npm version mismatch, npm transitive > dependency problems, proxy problem with older versions). All they could be > fixed but requires additional JS/NPM magic/knowledge. Without additional npm > build step the hdfs projects build could be kept more simple. > * The complexity of the planned SCM/KSM ui (hopefully it will remain simple) > doesn't require more sophisticated model. (Eg. we don't need JS require as we > need only a few controllers) > * HDFS developers mostly backend developers and not JS developers > 2. Frameworks > The big advantages of a more modern JS framework is the simplified > programming model (for example with two way databinding) I suggest to use a > more modern framework (not just jquery) which supports plain js (not just > ECMA2015/2016/typescript) and just include the required js files in the > projects (similar to the included bootstrap or as the existing namenode ui > works). > > * React could be a good candidate but it requires more library as it's just > a ui framework, even the REST calls need separated library. It could be used > with plain javascript instead of JSX and classes but not straightforward, and > it's more verbose. > > * Ember is used in yarnui2 but the main strength of the ember is the CLI > which couldn't be used for the simplified approach easily. I think ember is > best with the A.) option > * Angular 1 is a good candidate (but not so fancy). In case of angular 1 > the component based approach should be used (in that case later it could be > easier to migrate to angular 2 or react) > * The mainstream side of Angular 2 uses typescript, it could work with > plain JS but it could require additional knowledge, most of the tutorials and > documentation shows the typescript approach. > I suggest to use angular 1 or react. Maybe angular is easier to use as don't > need to emulate JSX with function calls, simple HTML templates could be used. > 3. Backend > I would prefer the approach of the existing namenode ui where the backend is > just the jmx endpoint. To keep it as simple as possible I suggest to try to > avoid dedicated REST backend if possible. Later we can use REST api of > SCM/KSM if they will be implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056630#comment-16056630
]
Chen Liang edited comment on HDFS-12002 at 6/20/17 11:15 PM:
-
Post initial patch.
Thanks [~elek] for the comments! Added the change you suggested.
was (Author: vagarychen):
Post initial patch.
Thanks [~elek] for the comments! Added the change you suggested. Also, I want
to take back the #4 change I mentioned in the description, as ozone is still
under development, it seems more beneficial to print the exception trace
message rather than hiding them for the time being.
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12002-HDFS-7240.001.patch
>
>
> Currently there are a few minor issues with the SCM CLI:
> 1. some commands do not use -c option to take container name. an issue with
> this is that arguments need to be in a certain order to be correctly parsed,
> e.g.:
> {{./bin/hdfs scm -container -del c0 -f}} works, but
> {{./bin/hdfs scm -container -del -f c0}} will not
> A more important thing is that, since -del requires the following argument
> being container name, if someone types {{./bin/hdfs scm -container -del
> -help}} it will be an error, while we probably want to display a help message
> instead.
> 2.some subcommands are not displaying the errors in the best way it could be,
> e.g.:
> {{./bin/hdfs scm -container -del}} is wrong because it misses container name.
> So cli complains
> {code}
> Missing argument for option: del
> Unrecognized options:[-container, -del]
> usage: hdfs scm []
> where can be one of the following
> -container Container related options
> {code}
> but this does not really show that it is container name it is missing
> 3. probably better to rename -del to -delete to be consistent with other
> commands like -create and -info
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12002:
--
Description:
Currently there are a few minor issues with the SCM CLI:
1. some commands do not use -c option to take container name. an issue with
this is that arguments need to be in a certain order to be correctly parsed,
e.g.:
{{./bin/hdfs scm -container -del c0 -f}} works, but
{{./bin/hdfs scm -container -del -f c0}} will not
A more important thing is that, since -del requires the following argument
being container name, if someone types {{./bin/hdfs scm -container -del -help}}
it will be an error, while we probably want to display a help message instead.
2.some subcommands are not displaying the errors in the best way it could be,
e.g.:
{{./bin/hdfs scm -container -del}} is wrong because it misses container name.
So cli complains
{code}
Missing argument for option: del
Unrecognized options:[-container, -del]
usage: hdfs scm []
where can be one of the following
-container Container related options
{code}
but this does not really show that it is container name it is missing
3. probably better to rename -del to -delete to be consistent with other
commands like -create and -info
was:
Currently there are a few minor issues with the SCM CLI:
1. some commands do not use -c option to take container name. an issue with
this is that arguments need to be in a certain order to be correctly parsed,
e.g.:
{{./bin/hdfs scm -container -del c0 -f}} works, but
{{./bin/hdfs scm -container -del -f c0}} will not
A more important thing is that, since -del requires the following argument
being container name, if someone types {{./bin/hdfs scm -container -del -help}}
it will be an error, while we probably want to display a help message instead.
2.some subcommands are not displaying the errors in the best way it could be,
e.g.:
{{./bin/hdfs scm -container -del}} is wrong because it misses container name.
So cli complains
{code}
Missing argument for option: del
Unrecognized options:[-container, -del]
usage: hdfs scm []
where can be one of the following
-container Container related options
{code}
but this does not really show that it is container name it is missing
3. probably better to rename -del to -delete to be consistent with other
commands like -create and -info
4. when passing in invalid argument e.g. -info on a non-existing container, an
exception will be displayed. We probably should not scare the users, and only
display just one error message. And move the exception display to debug mode
display or something.
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12002-HDFS-7240.001.patch
>
>
> Currently there are a few minor issues with the SCM CLI:
> 1. some commands do not use -c option to take container name. an issue with
> this is that arguments need to be in a certain order to be correctly parsed,
> e.g.:
> {{./bin/hdfs scm -container -del c0 -f}} works, but
> {{./bin/hdfs scm -container -del -f c0}} will not
> A more important thing is that, since -del requires the following argument
> being container name, if someone types {{./bin/hdfs scm -container -del
> -help}} it will be an error, while we probably want to display a help message
> instead.
> 2.some subcommands are not displaying the errors in the best way it could be,
> e.g.:
> {{./bin/hdfs scm -container -del}} is wrong because it misses container name.
> So cli complains
> {code}
> Missing argument for option: del
> Unrecognized options:[-container, -del]
> usage: hdfs scm []
> where can be one of the following
> -container Container related options
> {code}
> but this does not really show that it is container name it is missing
> 3. probably better to rename -del to -delete to be consistent with other
> commands like -create and -info
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11998) Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056637#comment-16056637
]
Hadoop QA commented on HDFS-11998:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestAvailableSpaceBlockPlacementPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestFileCorruption |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11998 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873727/HDFS-11998.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux e14040d2a258 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 45ff4d3 |
| Default Java | 1.8.0_131 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19970/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19970/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19970/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Enable DFSNetworkTopology as default
>
>
> Key: HDFS-11998
> URL: https://issues.apache.org/jira/browse/HDFS-11998
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11998.001.patch
>
>
> HDFS-11530 has made it configurable to use {{DFSNetworkTopology}}, and still
> uses {{NetworkTopology}} as
[jira] [Comment Edited] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056630#comment-16056630
]
Chen Liang edited comment on HDFS-12002 at 6/20/17 11:06 PM:
-
Post initial patch.
Thanks [~elek] for the comments! Added the change you suggested. Also, I want
to take back the #4 change I mentioned in the description, as ozone is still
under development, it seems more beneficial to print the exception trace
message rather than hiding them for the time being.
was (Author: vagarychen):
Post initial patch.
Thanks [~elek] for the comments! Added the change you suggested. Also, I want
to take back the #4 change I mentioned in the description, as ozone is still
under development, it seems more beneficial to print the error message rather
than hiding them for the time being.
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12002-HDFS-7240.001.patch
>
>
> Currently there are a few minor issues with the SCM CLI:
> 1. some commands do not use -c option to take container name. an issue with
> this is that arguments need to be in a certain order to be correctly parsed,
> e.g.:
> {{./bin/hdfs scm -container -del c0 -f}} works, but
> {{./bin/hdfs scm -container -del -f c0}} will not
> A more important thing is that, since -del requires the following argument
> being container name, if someone types {{./bin/hdfs scm -container -del
> -help}} it will be an error, while we probably want to display a help message
> instead.
> 2.some subcommands are not displaying the errors in the best way it could be,
> e.g.:
> {{./bin/hdfs scm -container -del}} is wrong because it misses container name.
> So cli complains
> {code}
> Missing argument for option: del
> Unrecognized options:[-container, -del]
> usage: hdfs scm []
> where can be one of the following
> -container Container related options
> {code}
> but this does not really show that it is container name it is missing
> 3. probably better to rename -del to -delete to be consistent with other
> commands like -create and -info
> 4. when passing in invalid argument e.g. -info on a non-existing container,
> an exception will be displayed. We probably should not scare the users, and
> only display just one error message. And move the exception display to debug
> mode display or something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12002:
--
Status: Patch Available (was: Open)
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12002-HDFS-7240.001.patch
>
>
> Currently there are a few minor issues with the SCM CLI:
> 1. some commands do not use -c option to take container name. an issue with
> this is that arguments need to be in a certain order to be correctly parsed,
> e.g.:
> {{./bin/hdfs scm -container -del c0 -f}} works, but
> {{./bin/hdfs scm -container -del -f c0}} will not
> A more important thing is that, since -del requires the following argument
> being container name, if someone types {{./bin/hdfs scm -container -del
> -help}} it will be an error, while we probably want to display a help message
> instead.
> 2.some subcommands are not displaying the errors in the best way it could be,
> e.g.:
> {{./bin/hdfs scm -container -del}} is wrong because it misses container name.
> So cli complains
> {code}
> Missing argument for option: del
> Unrecognized options:[-container, -del]
> usage: hdfs scm []
> where can be one of the following
> -container Container related options
> {code}
> but this does not really show that it is container name it is missing
> 3. probably better to rename -del to -delete to be consistent with other
> commands like -create and -info
> 4. when passing in invalid argument e.g. -info on a non-existing container,
> an exception will be displayed. We probably should not scare the users, and
> only display just one error message. And move the exception display to debug
> mode display or something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12002:
--
Attachment: HDFS-12002-HDFS-7240.001.patch
Post initial patch.
Thanks [~elek] for the comments! Added the change you suggested. Also, I want
to take back the #4 change I mentioned in the description, as ozone is still
under development, it seems more beneficial to print the error message rather
than hiding them for the time being.
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
> Attachments: HDFS-12002-HDFS-7240.001.patch
>
>
> Currently there are a few minor issues with the SCM CLI:
> 1. some commands do not use -c option to take container name. an issue with
> this is that arguments need to be in a certain order to be correctly parsed,
> e.g.:
> {{./bin/hdfs scm -container -del c0 -f}} works, but
> {{./bin/hdfs scm -container -del -f c0}} will not
> A more important thing is that, since -del requires the following argument
> being container name, if someone types {{./bin/hdfs scm -container -del
> -help}} it will be an error, while we probably want to display a help message
> instead.
> 2.some subcommands are not displaying the errors in the best way it could be,
> e.g.:
> {{./bin/hdfs scm -container -del}} is wrong because it misses container name.
> So cli complains
> {code}
> Missing argument for option: del
> Unrecognized options:[-container, -del]
> usage: hdfs scm []
> where can be one of the following
> -container Container related options
> {code}
> but this does not really show that it is container name it is missing
> 3. probably better to rename -del to -delete to be consistent with other
> commands like -create and -info
> 4. when passing in invalid argument e.g. -info on a non-existing container,
> an exception will be displayed. We probably should not scare the users, and
> only display just one error message. And move the exception display to debug
> mode display or something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12005) Ozone: Web interface for SCM
Elek, Marton created HDFS-12005: --- Summary: Ozone: Web interface for SCM Key: HDFS-12005 URL: https://issues.apache.org/jira/browse/HDFS-12005 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Elek, Marton Assignee: Elek, Marton This is a propsal about how a web interface could be implemented for SCM (and later for KSM) similar to the namenode ui. 1. JS framework There are three big option here. A.) One is to use a full featured web framework with all the webpack/npm minify/uglify magic. Build time the webpack/npm scripts should be run and the result will be added to the jar file. B.) It could be simplified if the generated minified/uglified js files are added to the project on commit time. It requires an additional step for every new patch (to generate the new minified javascripts) but doesn't require additional JS build tools during the build. C.) The third option is to make it as simple as possible similar to the current namenode ui which uses javascript but every dependency is commited (without JS minify/uglify and other preprocessing). I prefer to the third one as: * I have seen a lot of problems during frequent builds od older tez-ui versions (bower version mismatch, npm version mismatch, npm transitive dependency problems, proxy problem with older versions). All they could be fixed but requires additional JS/NPM magic/knowledge. Without additional npm build step the hdfs projects build could be kept more simple. * The complexity of the planned SCM/KSM ui (hopefully it will remain simple) doesn't require more sophisticated model. (Eg. we don't need JS require as we need only a few controllers) * HDFS developers mostly backend developers and not JS developers 2. Frameworks The big advantages of a more modern JS framework is the simplified programming model (for example with two way databinding) I suggest to use a more modern framework (not just jquery) which supports plain js (not just ECMA2015/2016/typescript) and just include the required js files in the projects (similar to the included bootstrap or as the existing namenode ui works). * React could be a good candidate but it requires more library as it's just a ui framework, even the REST calls need separated library. It could be used with plain javascript instead of JSX and classes but not straightforward, and it's more verbose. * Ember is used in yarnui2 but the main strength of the ember is the CLI which couldn't be used for the simplified approach easily. I think ember is best with the A.) option * Angular 1 is a good candidate (but not so fancy). In case of angular 1 the component based approach should be used (in that case later it could be easier to migrate to angular 2 or react) * The mainstream side of Angular 2 uses typescript, it could work with plain JS but it could require additional knowledge, most of the tutorials and documentation shows the typescript approach. I suggest to use angular 1 or react. Maybe angular is easier to use as don't need to emulate JSX with function calls, simple HTML templates could be used. 3. Backend I would prefer the approach of the existing namenode ui where the backend is just the jmx endpoint. To keep it as simple as possible I suggest to try to avoid dedicated REST backend if possible. Later we can use REST api of SCM/KSM if they will be implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11999) Ozone: Clarify startup error message of Datanode in case namenode is missing
[
https://issues.apache.org/jira/browse/HDFS-11999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056626#comment-16056626
]
Anu Engineer commented on HDFS-11999:
-
I will commit this shortly. There is a checkstyle warning issue I will fix it
while committing.
> Ozone: Clarify startup error message of Datanode in case namenode is missing
>
>
> Key: HDFS-11999
> URL: https://issues.apache.org/jira/browse/HDFS-11999
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-11999-HDFS-7240.001.patch, HDFS-11999.patch
>
>
> Datanode is failing if namenode config setting is missing even for Ozone with
> a confusing error message:
> {code}
> 14:33:29.176 [main] ERROR o.a.h.hdfs.server.datanode.DataNode - Exception in
> secureMain
> java.io.IOException: No services to connect (NameNodes or SCM).
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.refreshNamenodes(BlockPoolManager.java:168)
> ~[classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1440)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:510)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2802)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2705)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2752)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2896)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2920)
> [classes/:na]
> 14:33:29.177 [main] INFO org.apache.hadoop.util.ExitUtil - Exiting with
> status 1: java.io.IOException: No services to connect (NameNodes or SCM).
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11999) Ozone: Clarify startup error message of Datanode in case namenode is missing
[
https://issues.apache.org/jira/browse/HDFS-11999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056618#comment-16056618
]
Hadoop QA commented on HDFS-11999:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 48s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.balancer.TestBalancer |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | org.apache.hadoop.hdfs.security.token.block.TestBlockToken |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11999 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873728/HDFS-11999-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f8f7d88016a1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 185e3ad |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19971/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19971/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056613#comment-16056613
]
Elek, Marton commented on HDFS-12002:
-
+1: I suggest to print out the exception message in case of any error:
{code}
/../../hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/ozShell/Shell.java
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/ozShell/Shell.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/ozShell/Shell.java
index c0d3651b0bb..733fc6f7b3d 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/ozShell/Shell.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/ozShell/Shell.java
@@ -105,6 +105,8 @@ public static void main(String[] argv) throws Exception {
try {
res = ToolRunner.run(shell, argv);
} catch (Exception ex) {
+ System.err.println("ERROR: " + ex.getMessage());
System.exit(1);
}
System.exit(res);
{code}
I had no hadoop-site.xml (it's not required to run scm/ksm/namenode/datanode)
and currently without that the cli couldn't be started. (And no error message
at all).
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
>
> Currently there are a few minor issues with the SCM CLI:
> 1. some commands do not use -c option to take container name. an issue with
> this is that arguments need to be in a certain order to be correctly parsed,
> e.g.:
> {{./bin/hdfs scm -container -del c0 -f}} works, but
> {{./bin/hdfs scm -container -del -f c0}} will not
> A more important thing is that, since -del requires the following argument
> being container name, if someone types {{./bin/hdfs scm -container -del
> -help}} it will be an error, while we probably want to display a help message
> instead.
> 2.some subcommands are not displaying the errors in the best way it could be,
> e.g.:
> {{./bin/hdfs scm -container -del}} is wrong because it misses container name.
> So cli complains
> {code}
> Missing argument for option: del
> Unrecognized options:[-container, -del]
> usage: hdfs scm []
> where can be one of the following
> -container Container related options
> {code}
> but this does not really show that it is container name it is missing
> 3. probably better to rename -del to -delete to be consistent with other
> commands like -create and -info
> 4. when passing in invalid argument e.g. -info on a non-existing container,
> an exception will be displayed. We probably should not scare the users, and
> only display just one error message. And move the exception display to debug
> mode display or something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11881) NameNode consumes a lot of memory for snapshot diff report generation
[
https://issues.apache.org/jira/browse/HDFS-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056600#comment-16056600
]
Manoj Govindassamy commented on HDFS-11881:
---
Thanks for the review [~jojochuang].
bq. It looks to me that existing tests in TestSnapshotDiffReport cover most
code path. Maybe it's easier to update one of the test there to create 100 more
files.
TestSnapshotDiffReport is testing the diff report using HDFS API. But in the
context of the GC problem we are trying to solve here, we want the diff command
to be run over the shell. TestSnapshotCommands is already testing snapshot
commands over shell and looks like a better place for the new diff command test.
bq. have you done any kind of heap size measurement against a real cluster of
substantial size before/after this patch?
I was planning to do memory profiling for the broader fix. This particular jira
is more of short term, quick fix approach using the already existing and proven
methods. By looking at the code and comments, ChunkedArrayList is far better
than ArrayList in terms of memory usage. This quick fix might not sufficient to
solve all the GC problems around snapshot diff report, but definitely helps to
mitigate the problem.
bq. Diff#insert uses ArrayList to store created and deleted inodes. Considering
that a directory might have millions of created/deleted inodes in a snapshot,
there is a potential upside to convert these lists to ChunkedArrayList.
Thats right, DirectoryDiff list is still ArrayList and suffers from the
contiguous memory allocation issue. Will convert these to ChunkedArrayList.
> NameNode consumes a lot of memory for snapshot diff report generation
> -
>
> Key: HDFS-11881
> URL: https://issues.apache.org/jira/browse/HDFS-11881
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11881.01.patch
>
>
> *Problem:*
> HDFS supports a snapshot diff tool which can generate a [detailed report |
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html#Get_Snapshots_Difference_Report]
> of modified, created, deleted and renamed files between any 2 snapshots.
> {noformat}
> hdfs snapshotDiff
> {noformat}
> However, if the diff list between 2 snapshots happens to be huge, in the
> order of millions, then NameNode can consume a lot of memory while generating
> the huge diff report. In a few cases, we are seeing NameNode getting into a
> long GC lasting for few minutes to make room for this burst in memory
> requirement during snapshot diff report generation.
> *RootCause:*
> * NameNode tries to generate the diff report with all diff entries at once
> which puts undue pressure
> * Each diff report entry has the diff type (enum), source path byte array,
> and destination path byte array to the minimum. Let's take file deletions use
> case. For file deletions, there would be only source or destination paths in
> the diff report entry. Let's assume these deleted files on average take
> 128Bytes for the path. 4 million file deletion captured in diff report will
> thus need 512MB of memory
> * The snapshot diff report uses simple java ArrayList which tries to double
> its backing contiguous memory chunk every time the usage factor crosses the
> capacity threshold. So, a 512MB memory requirement might be internally asking
> for a much larger contiguous memory chunk
> *Proposal:*
> * Make NameNode snapshot diff report service follow the batch model (like
> directory listing service). Clients (hdfs snapshotDiff command) will then
> receive diff report in small batches, and need to iterate several times to
> get the full list.
> * Additionally, snap diff report service in the NameNode can make use of
> ChunkedArrayList data structure instead of the current ArrayList so as to
> avoid the curse of fragmentation and large contiguous memory requirement.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12004) Namenode UI continues to list DNs that have been removed from include and exclude
[ https://issues.apache.org/jira/browse/HDFS-12004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056598#comment-16056598 ] Erik Krogen commented on HDFS-12004: Ping [~tanping], [~wheat9], [~jingzhao], [~szetszwo] who were involved in those two previous JIRAs. > Namenode UI continues to list DNs that have been removed from include and > exclude > - > > Key: HDFS-12004 > URL: https://issues.apache.org/jira/browse/HDFS-12004 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Erik Krogen >Priority: Minor > > Initially in HDFS, after a DN was decommission and subsequently removed from > the exclude file (thus removing all references to it), it would still appear > in the NN UI as a "dead" node until the NN was restarted. In HDFS-1773, > discussion about this was had, and it was decided that the web UI should not > show these nodes. However when HDFS-5334 went through and the NN web UI was > reimplemented client-side, the behavior reverted back to pre-HDFS-1773, and > dead+decommissioned nodes once again showed in the dead list. This can be > operationally confusing for the same reasons as discussed in HDFS-1773. > I would like to open this discussion to determine if the regression was > intentional or if we should carry forward the logic implemented HDFS-1773 > into the new UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12004) Namenode UI continues to list DNs that have been removed from include and exclude
Erik Krogen created HDFS-12004: -- Summary: Namenode UI continues to list DNs that have been removed from include and exclude Key: HDFS-12004 URL: https://issues.apache.org/jira/browse/HDFS-12004 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.3.0 Reporter: Erik Krogen Priority: Minor Initially in HDFS, after a DN was decommission and subsequently removed from the exclude file (thus removing all references to it), it would still appear in the NN UI as a "dead" node until the NN was restarted. In HDFS-1773, discussion about this was had, and it was decided that the web UI should not show these nodes. However when HDFS-5334 went through and the NN web UI was reimplemented client-side, the behavior reverted back to pre-HDFS-1773, and dead+decommissioned nodes once again showed in the dead list. This can be operationally confusing for the same reasons as discussed in HDFS-1773. I would like to open this discussion to determine if the regression was intentional or if we should carry forward the logic implemented HDFS-1773 into the new UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[
https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056592#comment-16056592
]
Hadoop QA commented on HDFS-10702:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-10702 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10702 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846654/HDFS-10702.007.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19972/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add a Client API and Proxy Provider to enable stale read from Standby
> -
>
> Key: HDFS-10702
> URL: https://issues.apache.org/jira/browse/HDFS-10702
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Jiayi Zhou
>Assignee: Sean Mackrory
>Priority: Minor
> Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch,
> HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch,
> HDFS-10702.006.patch, HDFS-10702.007.patch, StaleReadfromStandbyNN.pdf
>
>
> Currently, clients must always talk to the active NameNode when performing
> any metadata operation, which means active NameNode could be a bottleneck for
> scalability. One way to solve this problem is to send read-only operations to
> Standby NameNode. The disadvantage is that it might be a stale read.
> Here, I'm thinking of adding a Client API to enable/disable stale read from
> Standby which gives Client the power to set the staleness restriction.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[ https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-11575: Status: Open (was: Patch Available) > Supporting HDFS NFS gateway with Federated HDFS > --- > > Key: HDFS-11575 > URL: https://issues.apache.org/jira/browse/HDFS-11575 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11575.001.patch, HDFS-11575.002.patch, > HDFS-11575.003.patch, HDFS-11575.004.patch, HDFS-11575.005.patch, > HDFS-11575.006.patch, HDFS-11575.007.patch, SupportingNFSwithFederatedHDFS.pdf > > > Currently HDFS NFS gateway only supports HDFS as the underlying filesystem. > Federated HDFS with ViewFS helps in improving the scalability of the name > nodes. However NFS is not supported with ViewFS. > With this change, ViewFS using HDFS as the underlying filesystem can be > exported using NFS. ViewFS mount table will be used to determine the exports > which needs to be supported. > Some important points > 1) This patch only supports HDFS as the underlying filesystem for ViewFS. > 2) This patch add support to add more than one export point in the NFS gateway > 3) Root filesystem of the ViewFS will not be mountable for NFS gateway with > ViewFS, >however this will not be the case for NFS gateway with HDFS > 4) A filehandle now apart from the field will also contain an identifier to > identify the name node, this will be used to map to correct name node for > file operations. > Please find the attached pdf document which helps in explaining the design > and the solution. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[ https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-11575: Status: Patch Available (was: Open) > Supporting HDFS NFS gateway with Federated HDFS > --- > > Key: HDFS-11575 > URL: https://issues.apache.org/jira/browse/HDFS-11575 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11575.001.patch, HDFS-11575.002.patch, > HDFS-11575.003.patch, HDFS-11575.004.patch, HDFS-11575.005.patch, > HDFS-11575.006.patch, HDFS-11575.007.patch, SupportingNFSwithFederatedHDFS.pdf > > > Currently HDFS NFS gateway only supports HDFS as the underlying filesystem. > Federated HDFS with ViewFS helps in improving the scalability of the name > nodes. However NFS is not supported with ViewFS. > With this change, ViewFS using HDFS as the underlying filesystem can be > exported using NFS. ViewFS mount table will be used to determine the exports > which needs to be supported. > Some important points > 1) This patch only supports HDFS as the underlying filesystem for ViewFS. > 2) This patch add support to add more than one export point in the NFS gateway > 3) Root filesystem of the ViewFS will not be mountable for NFS gateway with > ViewFS, >however this will not be the case for NFS gateway with HDFS > 4) A filehandle now apart from the field will also contain an identifier to > identify the name node, this will be used to map to correct name node for > file operations. > Please find the attached pdf document which helps in explaining the design > and the solution. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
[
https://issues.apache.org/jira/browse/HDFS-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12002:
--
Description:
Currently there are a few minor issues with the SCM CLI:
1. some commands do not use -c option to take container name. an issue with
this is that arguments need to be in a certain order to be correctly parsed,
e.g.:
{{./bin/hdfs scm -container -del c0 -f}} works, but
{{./bin/hdfs scm -container -del -f c0}} will not
A more important thing is that, since -del requires the following argument
being container name, if someone types {{./bin/hdfs scm -container -del -help}}
it will be an error, while we probably want to display a help message instead.
2.some subcommands are not displaying the errors in the best way it could be,
e.g.:
{{./bin/hdfs scm -container -del}} is wrong because it misses container name.
So cli complains
{code}
Missing argument for option: del
Unrecognized options:[-container, -del]
usage: hdfs scm []
where can be one of the following
-container Container related options
{code}
but this does not really show that it is container name it is missing
3. probably better to rename -del to -delete to be consistent with other
commands like -create and -info
4. when passing in invalid argument e.g. -info on a non-existing container, an
exception will be displayed. We probably should not scare the users, and only
display just one error message. And move the exception display to debug mode
display or something.
was:
Currently there are a few minor issues with the SCM CLI:
1. some commands do not use -c option to take container name. an issue with
this is that arguments need to be in a certain order to be correctly parsed,
e.g.:
{{./bin/hdfs scm -container -del c0 -f}} works, but
{{./bin/hdfs scm -container -del -f c0}} will not
2.some subcommands are not displaying the errors in the best way it could be,
e.g.:
{{./bin/hdfs scm -container -del}} is wrong because it misses container name.
So cli complains
{code}
Missing argument for option: del
Unrecognized options:[-container, -del]
usage: hdfs scm []
where can be one of the following
-container Container related options
{code}
but this does not really show that it is container name it is missing
3. probably better to rename -del to -delete to be consistent with other
commands like -create and -info
4. when passing in invalid argument e.g. -info on a non-existing container, an
exception will be displayed. We probably should not scare the users, and only
display just one error message. And move the exception display to debug mode
display or something.
> Ozone : SCM cli misc fixes/improvements
> ---
>
> Key: HDFS-12002
> URL: https://issues.apache.org/jira/browse/HDFS-12002
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
> Fix For: ozone
>
>
> Currently there are a few minor issues with the SCM CLI:
> 1. some commands do not use -c option to take container name. an issue with
> this is that arguments need to be in a certain order to be correctly parsed,
> e.g.:
> {{./bin/hdfs scm -container -del c0 -f}} works, but
> {{./bin/hdfs scm -container -del -f c0}} will not
> A more important thing is that, since -del requires the following argument
> being container name, if someone types {{./bin/hdfs scm -container -del
> -help}} it will be an error, while we probably want to display a help message
> instead.
> 2.some subcommands are not displaying the errors in the best way it could be,
> e.g.:
> {{./bin/hdfs scm -container -del}} is wrong because it misses container name.
> So cli complains
> {code}
> Missing argument for option: del
> Unrecognized options:[-container, -del]
> usage: hdfs scm []
> where can be one of the following
> -container Container related options
> {code}
> but this does not really show that it is container name it is missing
> 3. probably better to rename -del to -delete to be consistent with other
> commands like -create and -info
> 4. when passing in invalid argument e.g. -info on a non-existing container,
> an exception will be displayed. We probably should not scare the users, and
> only display just one error message. And move the exception display to debug
> mode display or something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory reassigned HDFS-10702: Assignee: Sean Mackrory (was: Jiayi Zhou) > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Sean Mackrory >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch, > HDFS-10702.006.patch, HDFS-10702.007.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11679) Ozone: SCM CLI: Implement list container command
[
https://issues.apache.org/jira/browse/HDFS-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056453#comment-16056453
]
Chen Liang commented on HDFS-11679:
---
Thanks [~yuanbo] for working on this, v003 patch looks pretty good to me.
One minor thing, seems that if I do {{./bin/hdfs scm -container -list}} or
{{./bin/hdfs scm -container -list -start XXX -prefix YYY}} (basically, without
specifying -count), it will display nothing. My understanding is that this is
because count is 0 by default, which is fine to me. But displaying nothing can
be somewhat confusing, I'm thinking that, when the output is going to be empty,
instead of showing nothing, we always display a message describing the possible
reasons for empty output, such as "need to specify -count, or prefix not exist,
or XYZ etc.".
Also if we list on an empty db (i.e. list when no container exist at all), we
get "Invalid start key, not found in current db." exception displayed. Maybe we
want to catch this and display something more informative (e.g. no container
exist).
> Ozone: SCM CLI: Implement list container command
>
>
> Key: HDFS-11679
> URL: https://issues.apache.org/jira/browse/HDFS-11679
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
> Labels: command-line
> Attachments: HDFS-11679-HDFS-7240.001.patch,
> HDFS-11679-HDFS-7240.002.patch, HDFS-11679-HDFS-7240.003.patch
>
>
> Implement the command to list containers
> {code}
> hdfs scm -container list -start [-count <100> | -end
> ]{code}
> Lists all containers known to SCM. The option -start allows the listing to
> start from a specified container and -count controls the number of entries
> returned but it is mutually exclusive with the -end option which returns keys
> from the -start to -end range.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11881) NameNode consumes a lot of memory for snapshot diff report generation
[
https://issues.apache.org/jira/browse/HDFS-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056447#comment-16056447
]
Wei-Chiu Chuang commented on HDFS-11881:
So for example, {{Diff#insert}} uses ArrayList to store created and deleted
inodes. Considering that a directory might have millions of created/deleted
inodes in a snapshot, there is a potential upside to convert these lists to
ChunkedArrayList. This is just a suggestion and I want to be cautious here: any
sort of heap usage optimization should be accompanied with a real cluster
benchmark.
> NameNode consumes a lot of memory for snapshot diff report generation
> -
>
> Key: HDFS-11881
> URL: https://issues.apache.org/jira/browse/HDFS-11881
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11881.01.patch
>
>
> *Problem:*
> HDFS supports a snapshot diff tool which can generate a [detailed report |
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html#Get_Snapshots_Difference_Report]
> of modified, created, deleted and renamed files between any 2 snapshots.
> {noformat}
> hdfs snapshotDiff
> {noformat}
> However, if the diff list between 2 snapshots happens to be huge, in the
> order of millions, then NameNode can consume a lot of memory while generating
> the huge diff report. In a few cases, we are seeing NameNode getting into a
> long GC lasting for few minutes to make room for this burst in memory
> requirement during snapshot diff report generation.
> *RootCause:*
> * NameNode tries to generate the diff report with all diff entries at once
> which puts undue pressure
> * Each diff report entry has the diff type (enum), source path byte array,
> and destination path byte array to the minimum. Let's take file deletions use
> case. For file deletions, there would be only source or destination paths in
> the diff report entry. Let's assume these deleted files on average take
> 128Bytes for the path. 4 million file deletion captured in diff report will
> thus need 512MB of memory
> * The snapshot diff report uses simple java ArrayList which tries to double
> its backing contiguous memory chunk every time the usage factor crosses the
> capacity threshold. So, a 512MB memory requirement might be internally asking
> for a much larger contiguous memory chunk
> *Proposal:*
> * Make NameNode snapshot diff report service follow the batch model (like
> directory listing service). Clients (hdfs snapshotDiff command) will then
> receive diff report in small batches, and need to iterate several times to
> get the full list.
> * Additionally, snap diff report service in the NameNode can make use of
> ChunkedArrayList data structure instead of the current ArrayList so as to
> avoid the curse of fragmentation and large contiguous memory requirement.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12003) Ozone: Misc : Cleanup error messages
Anu Engineer created HDFS-12003: --- Summary: Ozone: Misc : Cleanup error messages Key: HDFS-12003 URL: https://issues.apache.org/jira/browse/HDFS-12003 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Many error messages thrown from ozone are written for developers by developers. We need to review all publicly visible error messages to make sure it correct, includes enough context (stack traces do not count) and makes sense for the reader. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11782) Ozone: KSM: Add listKey
[ https://issues.apache.org/jira/browse/HDFS-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11782: Component/s: (was: HDFS-7240) ozone > Ozone: KSM: Add listKey > --- > > Key: HDFS-11782 > URL: https://issues.apache.org/jira/browse/HDFS-11782 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: ozone >Reporter: Anu Engineer >Assignee: Yiqun Lin > Fix For: HDFS-7240 > > Attachments: HDFS-11782-HDFS-7240.001.patch, > HDFS-11782-HDFS-7240.002.patch, HDFS-11782-HDFS-7240.003.patch, > HDFS-11782-HDFS-7240.004.patch > > > Add support for listing keys in a bucket. Just like other 2 list operations, > this API supports paging via, prevKey, prefix and maxKeys. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11873) Ozone: Object store handler cannot serve multiple requests from single http client
[
https://issues.apache.org/jira/browse/HDFS-11873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11873:
Component/s: (was: HDFS-7240)
ozone
> Ozone: Object store handler cannot serve multiple requests from single http
> client
> --
>
> Key: HDFS-11873
> URL: https://issues.apache.org/jira/browse/HDFS-11873
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Xiaoyu Yao
>Priority: Critical
> Attachments: HDFS-11873-HDFS-7240.testcase.patch
>
>
> This issue was found when I worked on HDFS-11846. Instead of creating a new
> http client instance per request, I tried to reuse {{CloseableHttpClient}} in
> {{OzoneClient}} class in a {{PoolingHttpClientConnectionManager}}. However,
> every second request from the http client hangs, which could not get
> dispatched to {{ObjectStoreJerseyContainer}}. There seems to be something
> wrong in the netty pipeline, this jira aims to 1) fix the problem in the
> server side 2) use the pool for client http clients to reduce the resource
> overhead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8457) Ozone: Refactor FsDatasetSpi to pull up HDFS-agnostic functionality into parent interface
[ https://issues.apache.org/jira/browse/HDFS-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-8457: --- Component/s: ozone > Ozone: Refactor FsDatasetSpi to pull up HDFS-agnostic functionality into > parent interface > - > > Key: HDFS-8457 > URL: https://issues.apache.org/jira/browse/HDFS-8457 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ozone >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-8457-HDFS-7240.01.patch, > HDFS-8457-HDFS-7240.02.patch, HDFS-8457-HDFS-7240.03.patch, > HDFS-8457-HDFS-7240.04.patch, HDFS-8457-HDFS-7240.05.patch, > HDFS-8457-HDFS-7240.06.patch, HDFS-8457-HDFS-7240.07.patch > > > FsDatasetSpi can be split up into HDFS-specific and HDFS-agnostic parts. The > HDFS-specific parts can continue to be retained in FsDataSpi while those > relating to volume management, block pools and upgrade can be moved to a > parent interface. > There will be no change to implementations of FsDatasetSpi. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8392) DataNode support for multiple datasets
[
https://issues.apache.org/jira/browse/HDFS-8392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-8392:
---
Component/s: ozone
> DataNode support for multiple datasets
> --
>
> Key: HDFS-8392
> URL: https://issues.apache.org/jira/browse/HDFS-8392
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ozone
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-8392-HDFS-7240.01.patch,
> HDFS-8392-HDFS-7240.02.patch, HDFS-8392-HDFS-7240.03.patch
>
>
> For HDFS-7240 we would like to share available DataNode storage across HDFS
> blocks and Ozone objects.
> The DataNode already supports sharing available storage across multiple block
> pool IDs for the federation feature. However all federated block pools use
> the same dataset implementation i.e. {{FsDatasetImpl}}.
> We can extend the DataNode to support multiple dataset implementations so the
> same storage space can be shared across one or more HDFS block pools and one
> or more Ozone block pools.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11881) NameNode consumes a lot of memory for snapshot diff report generation
[
https://issues.apache.org/jira/browse/HDFS-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056424#comment-16056424
]
Wei-Chiu Chuang commented on HDFS-11881:
I am not sure if it's necessary to add new tests. It looks to me that existing
tests in TestSnapshotDiffReport cover most code path. Maybe it's easier to
update one of the test there to create 100 more files.
Also, have you done any kind of heap size measurement against a real cluster of
substantial size before/after this patch?
I am not seeing the profile myself so I am hesitate to say changing the data
structures in these few places is sufficient to improve heap usage.
Thanks.
> NameNode consumes a lot of memory for snapshot diff report generation
> -
>
> Key: HDFS-11881
> URL: https://issues.apache.org/jira/browse/HDFS-11881
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11881.01.patch
>
>
> *Problem:*
> HDFS supports a snapshot diff tool which can generate a [detailed report |
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html#Get_Snapshots_Difference_Report]
> of modified, created, deleted and renamed files between any 2 snapshots.
> {noformat}
> hdfs snapshotDiff
> {noformat}
> However, if the diff list between 2 snapshots happens to be huge, in the
> order of millions, then NameNode can consume a lot of memory while generating
> the huge diff report. In a few cases, we are seeing NameNode getting into a
> long GC lasting for few minutes to make room for this burst in memory
> requirement during snapshot diff report generation.
> *RootCause:*
> * NameNode tries to generate the diff report with all diff entries at once
> which puts undue pressure
> * Each diff report entry has the diff type (enum), source path byte array,
> and destination path byte array to the minimum. Let's take file deletions use
> case. For file deletions, there would be only source or destination paths in
> the diff report entry. Let's assume these deleted files on average take
> 128Bytes for the path. 4 million file deletion captured in diff report will
> thus need 512MB of memory
> * The snapshot diff report uses simple java ArrayList which tries to double
> its backing contiguous memory chunk every time the usage factor crosses the
> capacity threshold. So, a 512MB memory requirement might be internally asking
> for a much larger contiguous memory chunk
> *Proposal:*
> * Make NameNode snapshot diff report service follow the batch model (like
> directory listing service). Clients (hdfs snapshotDiff command) will then
> receive diff report in small batches, and need to iterate several times to
> get the full list.
> * Additionally, snap diff report service in the NameNode can make use of
> ChunkedArrayList data structure instead of the current ArrayList so as to
> avoid the curse of fragmentation and large contiguous memory requirement.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11970) Ozone: TestXceiverClientManager.testFreeByEviction fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-11970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11970:
Issue Type: Sub-task (was: Bug)
Parent: HDFS-7240
> Ozone: TestXceiverClientManager.testFreeByEviction fails occasionally
> -
>
> Key: HDFS-11970
> URL: https://issues.apache.org/jira/browse/HDFS-11970
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-11970-HDFS-7240.001.patch
>
>
> TestXceiverClientManager.testFreeByEviction fails occasionally with the
> following stack trace.
> {code}
> Running org.apache.hadoop.ozone.scm.TestXceiverClientManager
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.989 sec <<<
> FAILURE! - in org.apache.hadoop.ozone.scm.TestXceiverClientManager
> testFreeByEviction(org.apache.hadoop.ozone.scm.TestXceiverClientManager)
> Time elapsed: 0.024 sec <<< FAILURE!
> java.lang.AssertionError:
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.ozone.scm.TestXceiverClientManager.testFreeByEviction(TestXceiverClientManager.java:184)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11991) Ozone: Ozone shell: the root is assumed to hdfs
[ https://issues.apache.org/jira/browse/HDFS-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056397#comment-16056397 ] Anu Engineer edited comment on HDFS-11991 at 6/20/17 8:26 PM: -- [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. 1. Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. 2. Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) 3. Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. bq. OzoneConfigKeys#OZONE_ADMINISTRATORS to get a list of administrators When we use REST protocol we might not have access to config file, if we decide to do it this way, then get config over rest might be a good idea. was (Author: anu): [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. 1. Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. 2. Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) 3. Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. > Ozone: Ozone shell: the root is assumed to hdfs > --- > > Key: HDFS-11991 > URL: https://issues.apache.org/jira/browse/HDFS-11991 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang > Fix For: HDFS-7240 > > > *hdfs oz* command, or ozone shell has a command like option to run some > commands as root easily by specifying _--root_ as a command line option. > But after HDFS-11655 that assumption is no longer true. We need to detect the > user that started the scm/ksm service and _root_ should be mapped to that > user. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11991) Ozone: Ozone shell: the root is assumed to hdfs
[ https://issues.apache.org/jira/browse/HDFS-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056397#comment-16056397 ] Anu Engineer edited comment on HDFS-11991 at 6/20/17 8:22 PM: -- [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. 1. Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. 2. Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) 3. Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. was (Author: anu): [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. # Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. # Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) # Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. > Ozone: Ozone shell: the root is assumed to hdfs > --- > > Key: HDFS-11991 > URL: https://issues.apache.org/jira/browse/HDFS-11991 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang > Fix For: HDFS-7240 > > > *hdfs oz* command, or ozone shell has a command like option to run some > commands as root easily by specifying _--root_ as a command line option. > But after HDFS-11655 that assumption is no longer true. We need to detect the > user that started the scm/ksm service and _root_ should be mapped to that > user. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11991) Ozone: Ozone shell: the root is assumed to hdfs
[ https://issues.apache.org/jira/browse/HDFS-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056397#comment-16056397 ] Anu Engineer edited comment on HDFS-11991 at 6/20/17 8:21 PM: -- [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. # Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. # Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) # Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. was (Author: anu): [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. # Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. # Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) # Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. > Ozone: Ozone shell: the root is assumed to hdfs > --- > > Key: HDFS-11991 > URL: https://issues.apache.org/jira/browse/HDFS-11991 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang > Fix For: HDFS-7240 > > > *hdfs oz* command, or ozone shell has a command like option to run some > commands as root easily by specifying _--root_ as a command line option. > But after HDFS-11655 that assumption is no longer true. We need to detect the > user that started the scm/ksm service and _root_ should be mapped to that > user. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11991) Ozone: Ozone shell: the root is assumed to hdfs
[ https://issues.apache.org/jira/browse/HDFS-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056397#comment-16056397 ] Anu Engineer edited comment on HDFS-11991 at 6/20/17 8:21 PM: -- [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. # Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. # Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) # Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. was (Author: anu): [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. # Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. # Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) # Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. > Ozone: Ozone shell: the root is assumed to hdfs > --- > > Key: HDFS-11991 > URL: https://issues.apache.org/jira/browse/HDFS-11991 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang > Fix For: HDFS-7240 > > > *hdfs oz* command, or ozone shell has a command like option to run some > commands as root easily by specifying _--root_ as a command line option. > But after HDFS-11655 that assumption is no longer true. We need to detect the > user that started the scm/ksm service and _root_ should be mapped to that > user. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11991) Ozone: Ozone shell: the root is assumed to hdfs
[ https://issues.apache.org/jira/browse/HDFS-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056397#comment-16056397 ] Anu Engineer commented on HDFS-11991: - [~cheersyang] Let us talk when you are back from vacation. There is a class of problems where reading information from the server leads to better decision on the client side. # Automatic Datanode Discovery - Problem: If we have a large cluster with many datanodes, the clients should be able to discover the various end-points that it can talk to, instead of talking to the same datanode all the time. This can be based on load or the block location. - Possible solutions : Rely on DNS to send this list of Datanodes, Virtual IP or the simplest one -- have KSM/SCM support an list of Datanodes and also support GetBlockLocation API over rest. # Discovery of Configs - Problem : There are times when a client can work more efficiently if it has access to the ozone-site.xml or the hadoop configs. - Solution : Support an API that returns configs -- There is an issue of security that we have solve though( avoiding man in the middle attacks) # Discovery of the Root user. - Problem : It is useful to know the name of the root user and group - Solution : Have an API return that from KSM/SCM. We might decide to build a discover API to get all these kinds of information from KSM/SCM. It will make it easy for the client to make more server friendly decision if we do that. In this specific case the discover API will start with the root user name. > Ozone: Ozone shell: the root is assumed to hdfs > --- > > Key: HDFS-11991 > URL: https://issues.apache.org/jira/browse/HDFS-11991 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang > Fix For: HDFS-7240 > > > *hdfs oz* command, or ozone shell has a command like option to run some > commands as root easily by specifying _--root_ as a command line option. > But after HDFS-11655 that assumption is no longer true. We need to detect the > user that started the scm/ksm service and _root_ should be mapped to that > user. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11983) Add documentation for metrics in KSMMetrics to OzoneMetrics.md
[ https://issues.apache.org/jira/browse/HDFS-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056380#comment-16056380 ] Anu Engineer commented on HDFS-11983: - +1, I am going to let you commit since you have the OzoneMetrics.md change also pending. Thought it might be better to let you commit in the order that you like. > Add documentation for metrics in KSMMetrics to OzoneMetrics.md > -- > > Key: HDFS-11983 > URL: https://issues.apache.org/jira/browse/HDFS-11983 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation, ozone >Affects Versions: HDFS-7240 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Attachments: HDFS-11983-HDFS-7240.001.patch, > HDFS-11983-HDFS-7240.002.patch > > > Metrics defined in KSMMetrics are not documented in OzoneMetrics.md. This > JIRA will track on this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11999) Ozone: Clarify startup error message of Datanode in case namenode is missing
[
https://issues.apache.org/jira/browse/HDFS-11999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-11999:
Attachment: HDFS-11999-HDFS-7240.001.patch
Renaming the patch into format that is needed by Jenkins to run against
HDFS-7240 branch.
> Ozone: Clarify startup error message of Datanode in case namenode is missing
>
>
> Key: HDFS-11999
> URL: https://issues.apache.org/jira/browse/HDFS-11999
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
> Attachments: HDFS-11999-HDFS-7240.001.patch, HDFS-11999.patch
>
>
> Datanode is failing if namenode config setting is missing even for Ozone with
> a confusing error message:
> {code}
> 14:33:29.176 [main] ERROR o.a.h.hdfs.server.datanode.DataNode - Exception in
> secureMain
> java.io.IOException: No services to connect (NameNodes or SCM).
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.refreshNamenodes(BlockPoolManager.java:168)
> ~[classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1440)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:510)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2802)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2705)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2752)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2896)
> [classes/:na]
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2920)
> [classes/:na]
> 14:33:29.177 [main] INFO org.apache.hadoop.util.ExitUtil - Exiting with
> status 1: java.io.IOException: No services to connect (NameNodes or SCM).
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11998) Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11998:
--
Attachment: HDFS-11998.001.patch
Post initial patch to set the config value to true by default.
> Enable DFSNetworkTopology as default
>
>
> Key: HDFS-11998
> URL: https://issues.apache.org/jira/browse/HDFS-11998
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11998.001.patch
>
>
> HDFS-11530 has made it configurable to use {{DFSNetworkTopology}}, and still
> uses {{NetworkTopology}} as default.
> Given the stress testing in HDFS-11923 which shows the correctness of
> DFSNetworkTopology, and the performance testing in HDFS-11535 which shows how
> DFSNetworkTopology can outperform NetworkTopology. I think we are at the
> point where I can and should enable DFSNetworkTopology as default.
> Any comments/thoughts are more than welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11998) Enable DFSNetworkTopology as default
[
https://issues.apache.org/jira/browse/HDFS-11998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11998:
--
Status: Patch Available (was: Open)
> Enable DFSNetworkTopology as default
>
>
> Key: HDFS-11998
> URL: https://issues.apache.org/jira/browse/HDFS-11998
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11998.001.patch
>
>
> HDFS-11530 has made it configurable to use {{DFSNetworkTopology}}, and still
> uses {{NetworkTopology}} as default.
> Given the stress testing in HDFS-11923 which shows the correctness of
> DFSNetworkTopology, and the performance testing in HDFS-11535 which shows how
> DFSNetworkTopology can outperform NetworkTopology. I think we are at the
> point where I can and should enable DFSNetworkTopology as default.
> Any comments/thoughts are more than welcome!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11881) NameNode consumes a lot of memory for snapshot diff report generation
[
https://issues.apache.org/jira/browse/HDFS-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056351#comment-16056351
]
Wei-Chiu Chuang commented on HDFS-11881:
Ah never mind... I saw your comment.
> NameNode consumes a lot of memory for snapshot diff report generation
> -
>
> Key: HDFS-11881
> URL: https://issues.apache.org/jira/browse/HDFS-11881
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11881.01.patch
>
>
> *Problem:*
> HDFS supports a snapshot diff tool which can generate a [detailed report |
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html#Get_Snapshots_Difference_Report]
> of modified, created, deleted and renamed files between any 2 snapshots.
> {noformat}
> hdfs snapshotDiff
> {noformat}
> However, if the diff list between 2 snapshots happens to be huge, in the
> order of millions, then NameNode can consume a lot of memory while generating
> the huge diff report. In a few cases, we are seeing NameNode getting into a
> long GC lasting for few minutes to make room for this burst in memory
> requirement during snapshot diff report generation.
> *RootCause:*
> * NameNode tries to generate the diff report with all diff entries at once
> which puts undue pressure
> * Each diff report entry has the diff type (enum), source path byte array,
> and destination path byte array to the minimum. Let's take file deletions use
> case. For file deletions, there would be only source or destination paths in
> the diff report entry. Let's assume these deleted files on average take
> 128Bytes for the path. 4 million file deletion captured in diff report will
> thus need 512MB of memory
> * The snapshot diff report uses simple java ArrayList which tries to double
> its backing contiguous memory chunk every time the usage factor crosses the
> capacity threshold. So, a 512MB memory requirement might be internally asking
> for a much larger contiguous memory chunk
> *Proposal:*
> * Make NameNode snapshot diff report service follow the batch model (like
> directory listing service). Clients (hdfs snapshotDiff command) will then
> receive diff report in small batches, and need to iterate several times to
> get the full list.
> * Additionally, snap diff report service in the NameNode can make use of
> ChunkedArrayList data structure instead of the current ArrayList so as to
> avoid the curse of fragmentation and large contiguous memory requirement.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12002) Ozone : SCM cli misc fixes/improvements
Chen Liang created HDFS-12002:
-
Summary: Ozone : SCM cli misc fixes/improvements
Key: HDFS-12002
URL: https://issues.apache.org/jira/browse/HDFS-12002
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Chen Liang
Assignee: Chen Liang
Fix For: ozone
Currently there are a few minor issues with the SCM CLI:
1. some commands do not use -c option to take container name. an issue with
this is that arguments need to be in a certain order to be correctly parsed,
e.g.:
{{./bin/hdfs scm -container -del c0 -f}} works, but
{{./bin/hdfs scm -container -del -f c0}} will not
2.some subcommands are not displaying the errors in the best way it could be,
e.g.:
{{./bin/hdfs scm -container -del}} is wrong because it misses container name.
So cli complains
{code}
Missing argument for option: del
Unrecognized options:[-container, -del]
usage: hdfs scm []
where can be one of the following
-container Container related options
{code}
but this does not really show that it is container name it is missing
3. probably better to rename -del to -delete to be consistent with other
commands like -create and -info
4. when passing in invalid argument e.g. -info on a non-existing container, an
exception will be displayed. We probably should not scare the users, and only
display just one error message. And move the exception display to debug mode
display or something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11971) libhdfs++: A few portability issues
[
https://issues.apache.org/jira/browse/HDFS-11971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-11971:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> libhdfs++: A few portability issues
> ---
>
> Key: HDFS-11971
> URL: https://issues.apache.org/jira/browse/HDFS-11971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-11971.HDFS-8707.000.patch,
> HDFS-11971.HDFS-8707.001.patch, HDFS-11971.HDFS-8707.002.patch,
> HDFS-11971.HDFS-8707.003.patch
>
>
> I recently encountered a few portability issues with libhdfs++ while trying
> to build it as a stand alone project (and also as part of another Apache
> project).
> 1. Method fixCase in configuration.h file produces a warning "conversion to
> ‘char’ from ‘int’ may alter its value [-Werror=conversion]" which does not
> allow libhdfs++ to be compiled as part of the codebase that treats such
> warnings as errors (can be fixed with a simple cast).
> 2. In CMakeLists.txt file (in libhdfspp directory) we do
> find_package(Threads) however we do not link it to the targets (e.g.
> hdfspp_static), which causes the build to fail with pthread errors. After the
> Threads package is found we need to link it using ${CMAKE_THREAD_LIBS_INIT}.
> 3. All the tools and examples fail to build as part of a standalone libhdfs++
> because they are missing multiple libraries such as protobuf, ssl, pthread,
> etc. This happens because we link them to a shared library hdfspp instead of
> hdfspp_static library. We should either link all the tools and examples to
> hdfspp_static library or explicitly add linking to all missing libraries for
> each tool/example.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11971) libhdfs++: A few portability issues
[
https://issues.apache.org/jira/browse/HDFS-11971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056313#comment-16056313
]
James Clampffer commented on HDFS-11971:
Thanks for the explanation and the extra fix for install paths
[~anatoli.shein]. +1, Committed to HDFS-8707.
> libhdfs++: A few portability issues
> ---
>
> Key: HDFS-11971
> URL: https://issues.apache.org/jira/browse/HDFS-11971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-11971.HDFS-8707.000.patch,
> HDFS-11971.HDFS-8707.001.patch, HDFS-11971.HDFS-8707.002.patch,
> HDFS-11971.HDFS-8707.003.patch
>
>
> I recently encountered a few portability issues with libhdfs++ while trying
> to build it as a stand alone project (and also as part of another Apache
> project).
> 1. Method fixCase in configuration.h file produces a warning "conversion to
> ‘char’ from ‘int’ may alter its value [-Werror=conversion]" which does not
> allow libhdfs++ to be compiled as part of the codebase that treats such
> warnings as errors (can be fixed with a simple cast).
> 2. In CMakeLists.txt file (in libhdfspp directory) we do
> find_package(Threads) however we do not link it to the targets (e.g.
> hdfspp_static), which causes the build to fail with pthread errors. After the
> Threads package is found we need to link it using ${CMAKE_THREAD_LIBS_INIT}.
> 3. All the tools and examples fail to build as part of a standalone libhdfs++
> because they are missing multiple libraries such as protobuf, ssl, pthread,
> etc. This happens because we link them to a shared library hdfspp instead of
> hdfspp_static library. We should either link all the tools and examples to
> hdfspp_static library or explicitly add linking to all missing libraries for
> each tool/example.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056311#comment-16056311 ] Hudson commented on HDFS-11647: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11897 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11897/]) HDFS-11647. Add -E option in hdfs "count" command to show erasure policy (lei: rev 45ff4d38e6175bc59b126633fc46927f8af9b641) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java * (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/cli/TestCLI.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Count.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Fix For: 3.0.0-alpha4 > > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch, HDFS-11647-007.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11647) Add -E option in hdfs "count" command to show erasure policy summarization
[ https://issues.apache.org/jira/browse/HDFS-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-11647: - Resolution: Fixed Fix Version/s: 3.0.0-alpha4 Status: Resolved (was: Patch Available) +1. Thanks for the contribution, [~luhuichun] Committed to trunk. > Add -E option in hdfs "count" command to show erasure policy summarization > -- > > Key: HDFS-11647 > URL: https://issues.apache.org/jira/browse/HDFS-11647 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: luhuichun > Labels: hdfs-ec-3.0-nice-to-have > Fix For: 3.0.0-alpha4 > > Attachments: HDFS-11647-001.patch, HDFS-11647-002.patch, > HDFS-11647-003.patch, HDFS-11647-004.patch, HDFS-11647-005.patch, > HDFS-11647-006.patch, HDFS-11647-007.patch > > > Add -E option in hdfs "count" command to show erasure policy summarization -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11518) libhdfs++: Add a build option to skip building examples, tests, and tools
[ https://issues.apache.org/jira/browse/HDFS-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11518: --- Resolution: Fixed Status: Resolved (was: Patch Available) > libhdfs++: Add a build option to skip building examples, tests, and tools > - > > Key: HDFS-11518 > URL: https://issues.apache.org/jira/browse/HDFS-11518 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11518.HDFS-8707.000.patch, > HDFS-11518.HDFS-8707.001.patch, HDFS-11518.HDFS-8707.002.patch > > > Adding a flag to just build the core library without tools, examples, and > tests will make it easier and lighter weight to embed the libhdfs++ source as > a third-party component of other projects. It won't need to look for a JDK, > valgrind, and gmock and won't generate a handful of binaries that might not > be relevant to other projects during normal use. > This should also make it a bit easier to wire into other build frameworks > since there won't be standalone binaries that need the path to other > libraries like protobuf while the library builds. They just need to be > around while the project embedding libhdfs++ gets linked. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11518) libhdfs++: Add a build option to skip building examples, tests, and tools
[ https://issues.apache.org/jira/browse/HDFS-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056269#comment-16056269 ] James Clampffer commented on HDFS-11518: +1, Thanks contributing the patch and including followup changes [~anatoli.shein]! Committed to HDFS-8707. > libhdfs++: Add a build option to skip building examples, tests, and tools > - > > Key: HDFS-11518 > URL: https://issues.apache.org/jira/browse/HDFS-11518 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11518.HDFS-8707.000.patch, > HDFS-11518.HDFS-8707.001.patch, HDFS-11518.HDFS-8707.002.patch > > > Adding a flag to just build the core library without tools, examples, and > tests will make it easier and lighter weight to embed the libhdfs++ source as > a third-party component of other projects. It won't need to look for a JDK, > valgrind, and gmock and won't generate a handful of binaries that might not > be relevant to other projects during normal use. > This should also make it a bit easier to wire into other build frameworks > since there won't be standalone binaries that need the path to other > libraries like protobuf while the library builds. They just need to be > around while the project embedding libhdfs++ gets linked. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11585) Ozone: Support force update a container
[ https://issues.apache.org/jira/browse/HDFS-11585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11585: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thank you for the contribution. I have committed this to the feature branch. > Ozone: Support force update a container > --- > > Key: HDFS-11585 > URL: https://issues.apache.org/jira/browse/HDFS-11585 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yuanbo Liu > Attachments: HDFS-11585-HDFS-7240.001.patch > > > HDFS-11567 added support of updating a container, and in following cases > # Container is closed > # Container meta file is falsely removed on disk or corrupted > a container cannot be gracefully updated. It is useful to support forcibly > update if a container gets into such state, that gives us the chance to > repair meta data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11585) Ozone: Support force update a container
[ https://issues.apache.org/jira/browse/HDFS-11585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056202#comment-16056202 ] Anu Engineer commented on HDFS-11585: - +1, I will commit this shortly. One small nit: In the tests we seem to have used "owner)" -- with a bracket, I am going to presume it is a typo and fix it while committing. > Ozone: Support force update a container > --- > > Key: HDFS-11585 > URL: https://issues.apache.org/jira/browse/HDFS-11585 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yuanbo Liu > Attachments: HDFS-11585-HDFS-7240.001.patch > > > HDFS-11567 added support of updating a container, and in following cases > # Container is closed > # Container meta file is falsely removed on disk or corrupted > a container cannot be gracefully updated. It is useful to support forcibly > update if a container gets into such state, that gives us the chance to > repair meta data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11963) Ozone: Documentation: Add getting started page
[ https://issues.apache.org/jira/browse/HDFS-11963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056188#comment-16056188 ] Anu Engineer commented on HDFS-11963: - +1, please feel free to commit. > Ozone: Documentation: Add getting started page > -- > > Key: HDFS-11963 > URL: https://issues.apache.org/jira/browse/HDFS-11963 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-11963-HDFS-7240.001.patch, > HDFS-11963-HDFS-7240.002.patch, HDFS-11963-HDFS-7240.003.patch, > HDFS-11963-HDFS-7240.004.patch, HDFS-11963-HDFS-7240.005.patch, > HDFS-11963-HDFS-7240.006.patch, HDFS-11963-HDFS-7240.addendum001.patch, > Screen Shot 2017-06-11 at 12.11.06 AM.png, Screen Shot 2017-06-11 at 12.11.19 > AM.png, Screen Shot 2017-06-11 at 12.11.32 AM.png > > > We need to add the Ozone section to hadoop documentation and also a section > on how to get started, that is what are configuration settings needed to > start running ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11960: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2 > > Attachments: HDFS-11960.patch, HDFS-11960-v2.branch-2.txt, > HDFS-11960-v2.trunk.txt > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-11960: -- Fix Version/s: 2.8.2 3.0.0-alpha4 2.9.0 > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2 > > Attachments: HDFS-11960.patch, HDFS-11960-v2.branch-2.txt, > HDFS-11960-v2.trunk.txt > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056111#comment-16056111 ] Kihwal Lee commented on HDFS-11960: --- HDFS-9754 was needed for the test. Cherry-picked HDFS-9754 and this jira to branch-2.8. The test passes now. > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2 > > Attachments: HDFS-11960.patch, HDFS-11960-v2.branch-2.txt, > HDFS-11960-v2.trunk.txt > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9754) Avoid unnecessary getBlockCollection calls in BlockManager
[
https://issues.apache.org/jira/browse/HDFS-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-9754:
-
Fix Version/s: 2.8.2
> Avoid unnecessary getBlockCollection calls in BlockManager
> --
>
> Key: HDFS-9754
> URL: https://issues.apache.org/jira/browse/HDFS-9754
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.9.0, 3.0.0-alpha1, 2.8.2
>
> Attachments: HDFS-9754.000.patch, HDFS-9754.001.patch,
> HDFS-9754.002.patch
>
>
> Currently BlockManager calls {{Namesystem#getBlockCollection}} in order to:
> 1. check if the block has already been abandoned
> 2. identify the storage policy of the block
> 3. meta save
> For #1 we can use BlockInfo's internal state instead of checking if the
> corresponding file still exists.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9754) Avoid unnecessary getBlockCollection calls in BlockManager
[
https://issues.apache.org/jira/browse/HDFS-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056107#comment-16056107
]
Kihwal Lee commented on HDFS-9754:
--
Cherry-picked to branch-2.8.
> Avoid unnecessary getBlockCollection calls in BlockManager
> --
>
> Key: HDFS-9754
> URL: https://issues.apache.org/jira/browse/HDFS-9754
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Fix For: 2.9.0, 3.0.0-alpha1, 2.8.2
>
> Attachments: HDFS-9754.000.patch, HDFS-9754.001.patch,
> HDFS-9754.002.patch
>
>
> Currently BlockManager calls {{Namesystem#getBlockCollection}} in order to:
> 1. check if the block has already been abandoned
> 2. identify the storage policy of the block
> 3. meta save
> For #1 we can use BlockInfo's internal state instead of checking if the
> corresponding file still exists.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11881) NameNode consumes a lot of memory for snapshot diff report generation
[
https://issues.apache.org/jira/browse/HDFS-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056091#comment-16056091
]
Wei-Chiu Chuang commented on HDFS-11881:
Hi Manoj, thanks for working on this patch. Good finding and looks like a good
improvement.
One quick question: it seems the scope of the patch is relatively limited to
your #2 (ChunkedArrayList in place of ArrayList). Do you plan to address your
#1 (report snapshot diff in batches)?
> NameNode consumes a lot of memory for snapshot diff report generation
> -
>
> Key: HDFS-11881
> URL: https://issues.apache.org/jira/browse/HDFS-11881
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11881.01.patch
>
>
> *Problem:*
> HDFS supports a snapshot diff tool which can generate a [detailed report |
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html#Get_Snapshots_Difference_Report]
> of modified, created, deleted and renamed files between any 2 snapshots.
> {noformat}
> hdfs snapshotDiff
> {noformat}
> However, if the diff list between 2 snapshots happens to be huge, in the
> order of millions, then NameNode can consume a lot of memory while generating
> the huge diff report. In a few cases, we are seeing NameNode getting into a
> long GC lasting for few minutes to make room for this burst in memory
> requirement during snapshot diff report generation.
> *RootCause:*
> * NameNode tries to generate the diff report with all diff entries at once
> which puts undue pressure
> * Each diff report entry has the diff type (enum), source path byte array,
> and destination path byte array to the minimum. Let's take file deletions use
> case. For file deletions, there would be only source or destination paths in
> the diff report entry. Let's assume these deleted files on average take
> 128Bytes for the path. 4 million file deletion captured in diff report will
> thus need 512MB of memory
> * The snapshot diff report uses simple java ArrayList which tries to double
> its backing contiguous memory chunk every time the usage factor crosses the
> capacity threshold. So, a 512MB memory requirement might be internally asking
> for a much larger contiguous memory chunk
> *Proposal:*
> * Make NameNode snapshot diff report service follow the batch model (like
> directory listing service). Clients (hdfs snapshotDiff command) will then
> receive diff report in small batches, and need to iterate several times to
> get the full list.
> * Additionally, snap diff report service in the NameNode can make use of
> ChunkedArrayList data structure instead of the current ArrayList so as to
> avoid the curse of fragmentation and large contiguous memory requirement.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs' in doc
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056085#comment-16056085 ] Hudson commented on HDFS-11978: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11896 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11896/]) HDFS-11978. Remove invalid '-usage' command of 'ec' and add missing (weichiu: rev 2b654a493cb88798bc5572b868ee1ffb411a07cb) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md * (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' in doc > -- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Assignee: wenxin he >Priority: Minor > Fix For: 3.0.0-alpha4 > > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs' in doc
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11978: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha4 Status: Resolved (was: Patch Available) Committed the patch to trunk. Thanks for contributing the patch. Would you please close the pull request? > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' in doc > -- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Assignee: wenxin he >Priority: Minor > Fix For: 3.0.0-alpha4 > > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs' in doc
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11978: --- Summary: Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs' in doc (was: Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs') > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' in doc > -- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Assignee: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11978) Remove invalid '-usage' command of 'ec' and add missing commands 'addPolicies' 'listCodecs'
[ https://issues.apache.org/jira/browse/HDFS-11978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056047#comment-16056047 ] Wei-Chiu Chuang commented on HDFS-11978: No, you are good. I will commit the patch shortly. > Remove invalid '-usage' command of 'ec' and add missing commands > 'addPolicies' 'listCodecs' > --- > > Key: HDFS-11978 > URL: https://issues.apache.org/jira/browse/HDFS-11978 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha4 >Reporter: wenxin he >Assignee: wenxin he >Priority: Minor > > Remove invalid '-usage' command of 'ec' in HDFSErasureCoding.md. > Add missing commands 'addPolicies' 'listCodecs' in HDFSCommands.md. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
[ https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-11575: - Attachment: HDFS-11575.007.patch > Supporting HDFS NFS gateway with Federated HDFS > --- > > Key: HDFS-11575 > URL: https://issues.apache.org/jira/browse/HDFS-11575 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11575.001.patch, HDFS-11575.002.patch, > HDFS-11575.003.patch, HDFS-11575.004.patch, HDFS-11575.005.patch, > HDFS-11575.006.patch, HDFS-11575.007.patch, SupportingNFSwithFederatedHDFS.pdf > > > Currently HDFS NFS gateway only supports HDFS as the underlying filesystem. > Federated HDFS with ViewFS helps in improving the scalability of the name > nodes. However NFS is not supported with ViewFS. > With this change, ViewFS using HDFS as the underlying filesystem can be > exported using NFS. ViewFS mount table will be used to determine the exports > which needs to be supported. > Some important points > 1) This patch only supports HDFS as the underlying filesystem for ViewFS. > 2) This patch add support to add more than one export point in the NFS gateway > 3) Root filesystem of the ViewFS will not be mountable for NFS gateway with > ViewFS, >however this will not be the case for NFS gateway with HDFS > 4) A filehandle now apart from the field will also contain an identifier to > identify the name node, this will be used to map to correct name node for > file operations. > Please find the attached pdf document which helps in explaining the design > and the solution. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11345) Document the configuration key for FSNamesystem lock fairness
[ https://issues.apache.org/jira/browse/HDFS-11345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055871#comment-16055871 ] Erik Krogen commented on HDFS-11345: Thanks [~ajisakaa]! > Document the configuration key for FSNamesystem lock fairness > - > > Key: HDFS-11345 > URL: https://issues.apache.org/jira/browse/HDFS-11345 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, namenode >Reporter: Zhe Zhang >Assignee: Erik Krogen >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha4, 2.8.2 > > Attachments: HADOOP-11345.000.patch, HADOOP-11345.001.patch, > HDFS-11345.002.patch > > > Per [earlier | > https://issues.apache.org/jira/browse/HDFS-5239?focusedCommentId=15536471=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15536471] > discussion. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055866#comment-16055866 ] Hudson commented on HDFS-11960: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11894 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11894/]) HDFS-11960. Successfully closed files can stay under-replicated. (kihwal: rev 8c0769dee4b455f4de08ccce36334f0be9e79e2c) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReconstruction.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11960.patch, HDFS-11960-v2.branch-2.txt, > HDFS-11960-v2.trunk.txt > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055843#comment-16055843 ] Kihwal Lee commented on HDFS-11960: --- Thanks for the review, Daryn. I've committed this to trunk and branch-2. On branch-2.8, the test fails due to an extra check in BlockManager. I will update the patch. > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11960.patch, HDFS-11960-v2.branch-2.txt, > HDFS-11960-v2.trunk.txt > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11989) Ozone: add TestKeysRatis and TestBucketsRatis
[
https://issues.apache.org/jira/browse/HDFS-11989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055820#comment-16055820
]
Hadoop QA commented on HDFS-11989:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
21s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestRatisManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11989 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873671/HDFS-11989-HDFS-7240.20170620c.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1b0c1ce4fe51 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / f8110c3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19968/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19968/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19968/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: add TestKeysRatis and TestBucketsRatis
> -
>
> Key: HDFS-11989
> URL: https://issues.apache.org/jira/browse/HDFS-11989
> Project: Hadoop HDFS
>
[jira] [Commented] (HDFS-11960) Successfully closed files can stay under-replicated.
[ https://issues.apache.org/jira/browse/HDFS-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055768#comment-16055768 ] Daryn Sharp commented on HDFS-11960: +1 Looks good. It's great to have finally solved this vexing problem with persistent under-replication! > Successfully closed files can stay under-replicated. > > > Key: HDFS-11960 > URL: https://issues.apache.org/jira/browse/HDFS-11960 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Attachments: HDFS-11960.patch, HDFS-11960-v2.branch-2.txt, > HDFS-11960-v2.trunk.txt > > > If a certain set of conditions hold at the time of a file creation, a block > of the file can stay under-replicated. This is because the block is > mistakenly taken out of the under-replicated block queue and never gets > reevaluated. > Re-evaluation can be triggered if > - a replica containing node dies. > - setrep is called > - NN repl queues are reinitialized (NN failover or restart) > If none of these happens, the block stays under-replicated. > Here is how it happens. > 1) A replica is finalized, but the ACK does not reach the upstream in time. > IBR is also delayed. > 2) A close recovery happens, which updates the gen stamp of "healthy" > replicas. > 3) The file is closed with the healthy replicas. It is added to the > replication queue. > 4) A replication is scheduled, so it is added to the pending replication > list. The replication target is picked as the failed node in 1). > 5) The old IBR is finally received for the failed/excluded node. In the > meantime, the replication fails, because there is already a finalized replica > (with older gen stamp) on the node. > 6) The IBR processing removes the block from the pending list, adds it to > corrupt replicas list, and then issues invalidation. Since the block is in > neither replication queue nor pending list, it stays under-replicated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[
https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055705#comment-16055705
]
Hadoop QA commented on HDFS-11943:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
37s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11943 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873655/HDFS-11943.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a78b6facc911 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 099dfe9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19967/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19967/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19967/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Warn log frequently print to screen in doEncode function on
>
[jira] [Commented] (HDFS-11992) Replace commons-logging APIs with slf4j in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055695#comment-16055695
]
Hadoop QA commented on HDFS-11992:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
35s{color} | {color:red} hadoop-common-project/hadoop-common in trunk has 19
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11992 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873622/HDFS-11992.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 21c91c5a373e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 099dfe9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19964/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/19964/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/19964/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/19964/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Updated] (HDFS-11989) Ozone: add TestKeysRatis and TestBucketsRatis
[ https://issues.apache.org/jira/browse/HDFS-11989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11989: --- Summary: Ozone: add TestKeysRatis and TestBucketsRatis (was: Ozone: add TestKeys with Ratis) Description: Add Ratis tests similar to TestKeys and TestBuckets. (was: Add a Ratis test similar to TestKeys.) > Ozone: add TestKeysRatis and TestBucketsRatis > - > > Key: HDFS-11989 > URL: https://issues.apache.org/jira/browse/HDFS-11989 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11989-HDFS-7240.20170618.patch, > HDFS-11989-HDFS-7240.20170620b.patch, HDFS-11989-HDFS-7240.20170620c.patch, > HDFS-11989-HDFS-7240.20170620.patch > > > Add Ratis tests similar to TestKeys and TestBuckets. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11989) Ozone: add TestKeys with Ratis
[ https://issues.apache.org/jira/browse/HDFS-11989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11989: --- Attachment: HDFS-11989-HDFS-7240.20170620c.patch HDFS-11989-HDFS-7240.20170620c.patch: adds also TestBucketsRatis. > Ozone: add TestKeys with Ratis > -- > > Key: HDFS-11989 > URL: https://issues.apache.org/jira/browse/HDFS-11989 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11989-HDFS-7240.20170618.patch, > HDFS-11989-HDFS-7240.20170620b.patch, HDFS-11989-HDFS-7240.20170620c.patch, > HDFS-11989-HDFS-7240.20170620.patch > > > Add a Ratis test similar to TestKeys. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055621#comment-16055621 ] liaoyuxiangqin commented on HDFS-11943: --- Thanks for your review and suggestions on this patch [~drankye]. As you suggestions the NativeXORRawEncoder will default use direct buffer for better performace. And use the PerformanceAdvisory LOG at debug don't affect performance, thanks. > Warn log frequently print to screen in doEncode function on > AbstractNativeRawEncoder class > --- > > Key: HDFS-11943 > URL: https://issues.apache.org/jira/browse/HDFS-11943 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, native >Affects Versions: 3.0.0-alpha4 > Environment: cluster: 3 nodes > os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, > Ubuntu4.4.0-31-generic) > hadoop version: hadoop-3.0.0-alpha4 > erasure coding: XOR-2-1-64k and enabled Intel ISA-L > hadoop fs -put file / >Reporter: liaoyuxiangqin >Assignee: liaoyuxiangqin >Priority: Minor > Attachments: HDFS-11943.002.patch, HDFS-11943.003.patch, > HDFS-11943.patch > > Original Estimate: 0.05h > Remaining Estimate: 0.05h > > when i write file to hdfs on above environment, the hdfs client frequently > print warn log of use direct ByteBuffer inputs/outputs in doEncode function > to screen, detail information as follows: > 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: > convertToByteBufferState is invoked, not efficiently. Please use direct > ByteBuffer inputs/outputs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11943) Warn log frequently print to screen in doEncode function on AbstractNativeRawEncoder class
[ https://issues.apache.org/jira/browse/HDFS-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liaoyuxiangqin updated HDFS-11943: -- Attachment: HDFS-11943.003.patch > Warn log frequently print to screen in doEncode function on > AbstractNativeRawEncoder class > --- > > Key: HDFS-11943 > URL: https://issues.apache.org/jira/browse/HDFS-11943 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, native >Affects Versions: 3.0.0-alpha4 > Environment: cluster: 3 nodes > os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, > Ubuntu4.4.0-31-generic) > hadoop version: hadoop-3.0.0-alpha4 > erasure coding: XOR-2-1-64k and enabled Intel ISA-L > hadoop fs -put file / >Reporter: liaoyuxiangqin >Assignee: liaoyuxiangqin >Priority: Minor > Attachments: HDFS-11943.002.patch, HDFS-11943.003.patch, > HDFS-11943.patch > > Original Estimate: 0.05h > Remaining Estimate: 0.05h > > when i write file to hdfs on above environment, the hdfs client frequently > print warn log of use direct ByteBuffer inputs/outputs in doEncode function > to screen, detail information as follows: > 2017-06-07 15:20:42,856 WARN rawcoder.AbstractNativeRawEncoder: > convertToByteBufferState is invoked, not efficiently. Please use direct > ByteBuffer inputs/outputs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
