[jira] [Updated] (HDFS-12054) FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12054: - Attachment: HDFS-12054.002.patch > FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode > --- > > Key: HDFS-12054 > URL: https://issues.apache.org/jira/browse/HDFS-12054 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch > > > In the process of FSNamesystem#addErasureCodingPolicies, it would be better > to call checkNameNodeSafeMode() to ensure NN is not in safemode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12054) FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12054: - Attachment: (was: HDFS-12054.002.patch) > FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode > --- > > Key: HDFS-12054 > URL: https://issues.apache.org/jira/browse/HDFS-12054 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch > > > In the process of FSNamesystem#addErasureCodingPolicies, it would be better > to call checkNameNodeSafeMode() to ensure NN is not in safemode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12054) FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12054: - Description: In the process of FSNamesystem#addErasureCodingPolicies, it would be better to call checkNameNodeSafeMode() to ensure NN is not in safemode. (was: In the process of FSNamesystem#addECPolicies, it would be better to call checkNameNodeSafeMode() to ensure NN is not in safemode.) Summary: FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode (was: FSNamesystem#addECPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode) > FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode > --- > > Key: HDFS-12054 > URL: https://issues.apache.org/jira/browse/HDFS-12054 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch > > > In the process of FSNamesystem#addErasureCodingPolicies, it would be better > to call checkNameNodeSafeMode() to ensure NN is not in safemode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121149#comment-16121149
]
Hadoop QA commented on HDFS-12274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
7s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
23s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881136/HDFS-12274-HDFS-7240.003.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux b4b7f05baa26 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personal
[jira] [Commented] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[
https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121110#comment-16121110
]
Kai Zheng commented on HDFS-11882:
--
Hi [~ajisakaa],
I'm trying to understand what you meant.
bq. When sentBytes is 18*64k and the cellSize is 64k,
The used schema should be 6 + 3. It sent 2 full strips of 2 x 9 x 64k bytes.
bq. DN1~8 will have two 64k data blocks, DN9~10 will have one 64k data block,
and DN11~14 will have two 64k parity blocks.
In the schema of 6 + 3, it only needs to write to 9 DNs. But here it looks like
to write DN1~14 or 14 DNs, why? I'm pretty confused here.
bq. In this situation, getNumAckedStriples() will return 2 if DN9 and DN10 are
failing.
OK, if it return 2 that means the acked strips number is 2, so the acked bytes
should be 2 x 9 x 64k, which should be right equal to the sent bytes.
bq. That way, in the testcase ackedBytes will become 20*64k, which is greater
than sentBytes.
Looks like 2 extra cells in addition to the 2 full strips also acked, since DN9
and DN10 are failing, they shouldn't contribute to the 2 extra acked cells.
I'm also trying to understand the root cause, why the acked bytes are greater
than the sent bytes. Cloud you help explain a little bit for me? Thanks!
> Client fails if acknowledged size is greater than bytes sent
>
>
> Key: HDFS-11882
> URL: https://issues.apache.org/jira/browse/HDFS-11882
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-11882.01.patch, HDFS-11882.02.patch,
> HDFS-11882.regressiontest.patch
>
>
> Some tests of erasure coding fails by the following exception. The following
> test was removed by HDFS-11823, however, this type of error can happen in
> real cluster.
> {noformat}
> Running
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure
> testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure)
> Time elapsed: 38.831 sec <<< ERROR!
> java.lang.IllegalStateException: null
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381)
> at
> org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121058#comment-16121058
]
Nandakumar commented on HDFS-12274:
---
Thanks [~cheersyang] for the suggestion, fixed it in patch v003.
> Ozone: Corona: move corona from test to tools package
> -
>
> Key: HDFS-12274
> URL: https://issues.apache.org/jira/browse/HDFS-12274
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12274-HDFS-7240.000.patch,
> HDFS-12274-HDFS-7240.001.patch, HDFS-12274-HDFS-7240.002.patch,
> HDFS-12274-HDFS-7240.003.patch
>
>
> This jira is to move {{Corona}} from test to tools package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12274:
--
Attachment: HDFS-12274-HDFS-7240.003.patch
> Ozone: Corona: move corona from test to tools package
> -
>
> Key: HDFS-12274
> URL: https://issues.apache.org/jira/browse/HDFS-12274
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12274-HDFS-7240.000.patch,
> HDFS-12274-HDFS-7240.001.patch, HDFS-12274-HDFS-7240.002.patch,
> HDFS-12274-HDFS-7240.003.patch
>
>
> This jira is to move {{Corona}} from test to tools package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12054) FSNamesystem#addECPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[
https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121002#comment-16121002
]
Hadoop QA commented on HDFS-12054:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HDFS-12054 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12054 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881126/HDFS-12054.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20628/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FSNamesystem#addECPolicies should call checkNameNodeSafeMode() to ensure
> Namenode is not in safemode
>
>
> Key: HDFS-12054
> URL: https://issues.apache.org/jira/browse/HDFS-12054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha3
>Reporter: lufei
>Assignee: lufei
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch
>
>
> In the process of FSNamesystem#addECPolicies, it would be better to call
> checkNameNodeSafeMode() to ensure NN is not in safemode.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12054) FSNamesystem#addECPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12054: - Attachment: HDFS-12054.002.patch > FSNamesystem#addECPolicies should call checkNameNodeSafeMode() to ensure > Namenode is not in safemode > > > Key: HDFS-12054 > URL: https://issues.apache.org/jira/browse/HDFS-12054 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch > > > In the process of FSNamesystem#addECPolicies, it would be better to call > checkNameNodeSafeMode() to ensure NN is not in safemode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12066) When Namenode is in safemode,may not allowed to remove an user's erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12066: - Attachment: (was: HDFS-12054.002.patch) > When Namenode is in safemode,may not allowed to remove an user's erasure > coding policy > -- > > Key: HDFS-12066 > URL: https://issues.apache.org/jira/browse/HDFS-12066 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12066.001.patch > > > FSNamesystem#removeErasureCodingPolicy should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12066) When Namenode is in safemode,may not allowed to remove an user's erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-12066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120998#comment-16120998
]
Hadoop QA commented on HDFS-12066:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-12066 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12066 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881125/HDFS-12054.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20627/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> When Namenode is in safemode,may not allowed to remove an user's erasure
> coding policy
> --
>
> Key: HDFS-12066
> URL: https://issues.apache.org/jira/browse/HDFS-12066
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha3
>Reporter: lufei
>Assignee: lufei
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12054.002.patch, HDFS-12066.001.patch
>
>
> FSNamesystem#removeErasureCodingPolicy should call checkNameNodeSafeMode() to
> ensure Namenode is not in safemode
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12066) When Namenode is in safemode,may not allowed to remove an user's erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12066: - Attachment: HDFS-12054.002.patch > When Namenode is in safemode,may not allowed to remove an user's erasure > coding policy > -- > > Key: HDFS-12066 > URL: https://issues.apache.org/jira/browse/HDFS-12066 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.002.patch, HDFS-12066.001.patch > > > FSNamesystem#removeErasureCodingPolicy should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6489) DFS Used space is not correct computed on frequent append operations
[
https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120964#comment-16120964
]
Weiwei Yang commented on HDFS-6489:
---
I am un-assigning myself from this ticket, [~raviprak] do you want to take this
over?
> DFS Used space is not correct computed on frequent append operations
>
>
> Key: HDFS-6489
> URL: https://issues.apache.org/jira/browse/HDFS-6489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0, 2.7.1, 2.7.2
>Reporter: stanley shi
>Assignee: Weiwei Yang
> Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch,
> HDFS-6489.003.patch, HDFS-6489.004.patch, HDFS-6489.005.patch,
> HDFS-6489.006.patch, HDFS-6489.007.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space
> on each block write operation. This is correct in most scenario (create new
> file), but sometimes it will behave in-correct(append small data to a large
> block).
> For example, I have a file with only one block(say, 60M). Then I try to
> append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the
> block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large
> number of files (1000+), assume the block size is 32M (half of the default
> value), then the dfs used will be increased 1000*32M = 32G on each append to
> the files; but actually I only write 10K bytes; this will cause the datanode
> to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
> BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received
> exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException:
> Insufficient space for appending to FinalizedReplica, blk_1073742834_45306,
> FINALIZED{quote}
> But the actual disk usage:
> {quote}
> [root@hdsh143 ~]# df -h
> FilesystemSize Used Avail Use% Mounted on
> /dev/sda3 16G 2.9G 13G 20% /
> tmpfs 1.9G 72K 1.9G 1% /dev/shm
> /dev/sda1 97M 32M 61M 35% /boot
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-6489) DFS Used space is not correct computed on frequent append operations
[
https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang reassigned HDFS-6489:
-
Assignee: (was: Weiwei Yang)
> DFS Used space is not correct computed on frequent append operations
>
>
> Key: HDFS-6489
> URL: https://issues.apache.org/jira/browse/HDFS-6489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0, 2.7.1, 2.7.2
>Reporter: stanley shi
> Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch,
> HDFS-6489.003.patch, HDFS-6489.004.patch, HDFS-6489.005.patch,
> HDFS-6489.006.patch, HDFS-6489.007.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space
> on each block write operation. This is correct in most scenario (create new
> file), but sometimes it will behave in-correct(append small data to a large
> block).
> For example, I have a file with only one block(say, 60M). Then I try to
> append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the
> block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large
> number of files (1000+), assume the block size is 32M (half of the default
> value), then the dfs used will be increased 1000*32M = 32G on each append to
> the files; but actually I only write 10K bytes; this will cause the datanode
> to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
> BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received
> exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException:
> Insufficient space for appending to FinalizedReplica, blk_1073742834_45306,
> FINALIZED{quote}
> But the actual disk usage:
> {quote}
> [root@hdsh143 ~]# df -h
> FilesystemSize Used Avail Use% Mounted on
> /dev/sda3 16G 2.9G 13G 20% /
> tmpfs 1.9G 72K 1.9G 1% /dev/shm
> /dev/sda1 97M 32M 61M 35% /boot
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120908#comment-16120908
]
Weiwei Yang commented on HDFS-12274:
The patch looks good to me, only one thing, following code
{code}
+byte[] value = RandomStringUtils.randomAscii(10240)
+.getBytes(Charset.forName("UTF-8"));
{code}
uses hard coded charset, can we replace this with {{DFSUtil.string2Bytes()}} ?
Thanks
> Ozone: Corona: move corona from test to tools package
> -
>
> Key: HDFS-12274
> URL: https://issues.apache.org/jira/browse/HDFS-12274
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12274-HDFS-7240.000.patch,
> HDFS-12274-HDFS-7240.001.patch, HDFS-12274-HDFS-7240.002.patch
>
>
> This jira is to move {{Corona}} from test to tools package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120906#comment-16120906 ] Weiwei Yang commented on HDFS-12281: Hi [~ajayydv] thanks for working on this, I am OK with the first 2 changes, but can we use RocksDB as the default in java class to be consistent with ozone-default.xml ? I think RocksDB should be the default implementation for metastore. > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > Attachments: HDFS-12281-HDFS-7240.01.patch > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12202) Provide new set of FileSystem API to bypass external attribute provider
[
https://issues.apache.org/jira/browse/HDFS-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120861#comment-16120861
]
Chris Douglas commented on HDFS-12202:
--
bq. I wish there is a solution that we can avoid modifying HDFS API
Modifying the FileSystem API to support this single, narrow use case is not a
good tradeoff. Let's find some reasonable workaround(s).
bq. the same user may run different applications too than distcp too
bq. ANY user can run distcp, and distcp can happen within a same cluster too.
If we want to these, would it be too restrictive.
Is it so restrictive? In trunk, the new shell scripts can swap out distcp with
another utility that could delegate the request to a service. Running a
dedicated service for a deployment with this _very_ specific constraint is not
so dire.
bq. we have the problem of not knowing when to pass through, because only users
knows when to pass through and we don't have a way to fill the gap between user
(accessing FileSystem API only) and the pass through API of external attribute
provider
Is filtering sufficient? Or does this also need to add attributes that the
external attribute provider strips out? If distcp only needs to filter out some
extended attributes, then the client can do this without cooperation from HDFS.
> Provide new set of FileSystem API to bypass external attribute provider
> ---
>
> Key: HDFS-12202
> URL: https://issues.apache.org/jira/browse/HDFS-12202
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs, hdfs-client
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> HDFS client uses
> {code}
> /**
>* Return a file status object that represents the path.
>* @param f The path we want information from
>* @return a FileStatus object
>* @throws FileNotFoundException when the path does not exist
>* @throws IOException see specific implementation
>*/
> public abstract FileStatus getFileStatus(Path f) throws IOException;
> /**
>* List the statuses of the files/directories in the given path if the path
> is
>* a directory.
>*
>* Does not guarantee to return the List of files/directories status in a
>* sorted order.
>*
>* Will not return null. Expect IOException upon access error.
>* @param f given path
>* @return the statuses of the files/directories in the given patch
>* @throws FileNotFoundException when the path does not exist
>* @throws IOException see specific implementation
>*/
> public abstract FileStatus[] listStatus(Path f) throws
> FileNotFoundException,
> IOException;
> {code}
> to get FileStatus of files.
> When external attribute provider (INodeAttributeProvider) is enabled for a
> cluster, the external attribute provider is consulted to get back some
> relevant info (including ACL, group etc) and returned back in FileStatus,
> There is a problem here, when we use distcp to copy files from srcCluster to
> tgtCluster, if srcCluster has external attribute provider enabled, the data
> we copied would contain data from attribute provider, which we may not want.
> Create this jira to add a new set of interface for distcp to use, so that
> distcp can copy HDFS data only and bypass external attribute provider data.
> The new set API would look like
> {code}
> /**
>* Return a file status object that represents the path.
>* @param f The path we want information from
>* @param bypassExtAttrProvider if true, bypass external attr provider
>*when it's in use.
>* @return a FileStatus object
>* @throws FileNotFoundException when the path does not exist
>* @throws IOException see specific implementation
>*/
> public FileStatus getFileStatus(Path f,
> final boolean bypassExtAttrProvider) throws IOException;
> /**
>* List the statuses of the files/directories in the given path if the path
> is
>* a directory.
>*
>* Does not guarantee to return the List of files/directories status in a
>* sorted order.
>*
>* Will not return null. Expect IOException upon access error.
>* @param f
>* @param bypassExtAttrProvider if true, bypass external attr provider
>*when it's in use.
>* @return
>* @throws FileNotFoundException
>* @throws IOException
>*/
> public FileStatus[] listStatus(Path f,
> final boolean bypassExtAttrProvider) throws FileNotFoundException,
> IOException;
> {code}
> So when bypassExtAttrProvider is true, external attribute provider will be
> bypassed.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120838#comment-16120838 ] Anu Engineer commented on HDFS-12159: - [~szetszwo] Can you please take a look at the changes made to XcieverRatisServer ? > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: createFlow.png, HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120836#comment-16120836 ] Anu Engineer edited comment on HDFS-12159 at 8/9/17 11:38 PM: -- This patch brings in Apache Ratis as a first class pipeline for Ozone Containers. The next patch will have client and test changes where we can run the same tests against Ratis as well as Stand alone pipelines. This patch also addresses all comments from earlier reviews. was (Author: anu): This patch brings in Apache Ratis as a first class pipeline for Ozone Containers. The next patch will have client and test changes where we can run the same tests against Ratis as well as Stand alone pipelines. > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: createFlow.png, HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12159: Attachment: createFlow.png This PNG explains the create workflow with Ratis or pluggable pipeline support > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: createFlow.png, HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12159: Attachment: HDFS-12159-HDFS-7240.003.patch This patch brings in Apache Ratis as a first class pipeline for Ozone Containers. The next patch will have client and test changes where we can run the same tests against Ratis as well as Stand alone pipelines. > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12000) Ozone: Container : Add key versioning support-1
[
https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120823#comment-16120823
]
Hadoop QA commented on HDFS-12000:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 38s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.cblock.TestBufferManager |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12000 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881089/HDFS-12000-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 83ab5bf5844e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HD
[jira] [Commented] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120808#comment-16120808
]
Hadoop QA commented on HDFS-12278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
53s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 52m
11s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_131.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_144 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.server.namenode.TestStartup |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:d946387 |
| JIRA Issue | HDFS-12278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881080/HDFS-12278-branch-2.8.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 060b17dc6b37 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess
[jira] [Commented] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[
https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120776#comment-16120776
]
Hadoop QA commented on HDFS-12281:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.cblock.TestBufferManager |
| | hadoop.ozone.web.client.TestKeys |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12281 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881086/HDFS-12281-HDFS-7240.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 0f248962b2a3 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / c25d959 |
| Default Java | 1.8.0_131 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20624/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20624/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20624/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: Ozone-default.xml has 3 properties that do not match the default
> Config value
>
>
> Key: HDFS-12281
> URL: https://issues.apache.org/jira/browse/HDFS-12281
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Ajay Yadav
>Assignee: Ajay Yadav
> Fix For: HDFS-7240
>
> Attachments: HDFS-122
[jira] [Updated] (HDFS-12285) Better handling of namenode ip address change
[
https://issues.apache.org/jira/browse/HDFS-12285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-12285:
---
Description:
RPC client layer provides functionality to detect ip address change:
{noformat}
Client.java
private synchronized boolean updateAddress() throws IOException {
// Do a fresh lookup with the old host name.
InetSocketAddress currentAddr = NetUtils.createSocketAddrForHost(
server.getHostName(), server.getPort());
..
}
{noformat}
To use this feature, we need to enable retry via
{{dfs.client.retry.policy.enabled}}. Otherwise {{TryOnceThenFail}} RetryPolicy
will be used; which caused {{handleConnectionFailure}} to throw
{{ConnectException}} exception without retrying with the new ip address.
{noformat}
private void handleConnectionFailure(int curRetries, IOException ioe
) throws IOException {
closeConnection();
final RetryAction action;
try {
action = connectionRetryPolicy.shouldRetry(ioe, curRetries, 0, true);
} catch(Exception e) {
throw e instanceof IOException? (IOException)e: new IOException(e);
}
..
}
{noformat}
However, using such configuration isn't ideal. What happens is DFSClient still
holds onto the cached old ip address created by {{namenode =
proxyInfo.getProxy();}}. Thus when a new rpc connection is created, it starts
with the old ip followed by retry with the new ip. It will be nice if DFSClient
can update namenode proxy automatically upon ip address change.
was:
RPC client layer provides functionality to detect ip address change:
{noformat}
Client.java
private synchronized boolean updateAddress() throws IOException {
// Do a fresh lookup with the old host name.
InetSocketAddress currentAddr = NetUtils.createSocketAddrForHost(
server.getHostName(), server.getPort());
..
}
{noformat}
To use this feature, we need to enable retry via
{{dfs.client.retry.policy.enabled}}. Otherwise {{TryOnceThenFail}} RetryPolicy
will be used; which caused {{handleConnectionFailure}} to throw
{{ConnectException}} exception without retrying with the new ip address.
{noformat}
private void handleConnectionFailure(int curRetries, IOException ioe
) throws IOException {
closeConnection();
final RetryAction action;
try {
action = connectionRetryPolicy.shouldRetry(ioe, curRetries, 0, true);
} catch(Exception e) {
throw e instanceof IOException? (IOException)e: new IOException(e);
}
..
}
{noformat}
However, using such configuration isn't ideal. What happens is DFSClient still
has the cached the old ip address created by {{namenode =
proxyInfo.getProxy();}}. Then when a new rpc connection is created, it starts
with the old ip followed by retry with the new ip. It will be nice if DFSClient
can refresh namenode proxy automatically.
> Better handling of namenode ip address change
> -
>
> Key: HDFS-12285
> URL: https://issues.apache.org/jira/browse/HDFS-12285
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>
> RPC client layer provides functionality to detect ip address change:
> {noformat}
> Client.java
> private synchronized boolean updateAddress() throws IOException {
> // Do a fresh lookup with the old host name.
> InetSocketAddress currentAddr = NetUtils.createSocketAddrForHost(
>server.getHostName(), server.getPort());
> ..
> }
> {noformat}
> To use this feature, we need to enable retry via
> {{dfs.client.retry.policy.enabled}}. Otherwise {{TryOnceThenFail}}
> RetryPolicy will be used; which caused {{handleConnectionFailure}} to throw
> {{ConnectException}} exception without retrying with the new ip address.
> {noformat}
> private void handleConnectionFailure(int curRetries, IOException ioe
> ) throws IOException {
> closeConnection();
> final RetryAction action;
> try {
> action = connectionRetryPolicy.shouldRetry(ioe, curRetries, 0, true);
> } catch(Exception e) {
> throw e instanceof IOException? (IOException)e: new IOException(e);
> }
> ..
> }
> {noformat}
> However, using such configuration isn't ideal. What happens is DFSClient
> still holds onto the cached old ip address created by {{namenode =
> proxyInfo.getProxy();}}. Thus when a new rpc connection is created, it starts
> with the old ip followed by retry with the new ip. It will be nice if
> DFSClient can update namenode proxy automatically upon ip address change.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@ha
[jira] [Assigned] (HDFS-12102) VolumeScanner throttle dropped (fast scan enabled) when there is a corrupt block
[ https://issues.apache.org/jira/browse/HDFS-12102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDFS-12102: Assignee: Ashwin Ramesh > VolumeScanner throttle dropped (fast scan enabled) when there is a corrupt > block > > > Key: HDFS-12102 > URL: https://issues.apache.org/jira/browse/HDFS-12102 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Affects Versions: 2.8.2 >Reporter: Ashwin Ramesh >Assignee: Ashwin Ramesh >Priority: Minor > Fix For: 2.8.2 > > Attachments: HDFS-12102-001.patch, HDFS-12102-002.patch, > HDFS-12102-003.patch > > > When the Volume scanner sees a corrupt block, it restarts the scan and scans > the blocks at much faster rate with a negligible scan period. This is so that > it doesn't take 3 weeks to report blocks since a corrupt block means > increased likelihood that there are more corrupt blocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12285) Better handling of namenode ip address change
Ming Ma created HDFS-12285:
--
Summary: Better handling of namenode ip address change
Key: HDFS-12285
URL: https://issues.apache.org/jira/browse/HDFS-12285
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ming Ma
RPC client layer provides functionality to detect ip address change:
{noformat}
Client.java
private synchronized boolean updateAddress() throws IOException {
// Do a fresh lookup with the old host name.
InetSocketAddress currentAddr = NetUtils.createSocketAddrForHost(
server.getHostName(), server.getPort());
..
}
{noformat}
To use this feature, we need to enable retry via
{{dfs.client.retry.policy.enabled}}. Otherwise {{TryOnceThenFail}} RetryPolicy
will be used; which caused {{handleConnectionFailure}} to throw
{{ConnectException}} exception without retrying with the new ip address.
{noformat}
private void handleConnectionFailure(int curRetries, IOException ioe
) throws IOException {
closeConnection();
final RetryAction action;
try {
action = connectionRetryPolicy.shouldRetry(ioe, curRetries, 0, true);
} catch(Exception e) {
throw e instanceof IOException? (IOException)e: new IOException(e);
}
..
}
{noformat}
However, using such configuration isn't ideal. What happens is DFSClient still
has the cached the old ip address created by {{namenode =
proxyInfo.getProxy();}}. Then when a new rpc connection is created, it starts
with the old ip followed by retry with the new ip. It will be nice if DFSClient
can refresh namenode proxy automatically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120729#comment-16120729
]
Hudson commented on HDFS-12278:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12156 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12156/])
HDFS-12278. LeaseManager operations are inefficient in 2.8. Contributed
(kihwal: rev b5c02f95b5a2fcb8931d4a86f8192caa18009ea9)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120726#comment-16120726
]
Rushabh S Shah commented on HDFS-12278:
---
Thanks [~kihwal] for the review and commit !
Thanks [~daryn] for the benchmarking the change.
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-12278:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.2
3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
I've committed this to trunk, branch-2, branch-2.8 and branch-2.8.2. Thanks for
reporting and fixing the issue.
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120702#comment-16120702
]
Rushabh S Shah edited comment on HDFS-12278 at 8/9/17 9:46 PM:
---
bq. but how did you benchmark TreeSet and PriorityQueue
[~daryn] benchmarked.
He just created lease-like objects and tested renew like methods.
Basically an object with string, int member variable and a comparator.
He created 100,000 such objects and called renew on them and measured via
{{monotonicTime}}.
Daryn: please correct me if I am wrong.
bq. Are you aware of JMH?
I wasn't aware until I read the comment and did web search. Thanks!
But it is very simple to understand that priority queue is not a good data
structure if you want to remove any object other than the top one.
was (Author: shahrs87):
bq. but how did you benchmark TreeSet and PriorityQueue
[~daryn] benchmarked.
He just created lease-like objects and tested renew like methods.
Basically an object with string, int member variable and a comparator.
He created 100,000 such objects and called renew on them and measured via
{{monotonicTime}}.
Daryn: please correct me if I am wrong.
bq. Are you aware of JMH?
I wasn't aware until I read the comment as did web search.
But it is very simple to understand that priority queue is not a good data
structure if you want to remove any object other than the top one.
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120702#comment-16120702
]
Rushabh S Shah commented on HDFS-12278:
---
bq. but how did you benchmark TreeSet and PriorityQueue
[~daryn] benchmarked.
He just created lease-like objects and tested renew like methods.
Basically an object with string, int member variable and a comparator.
He created 100,000 such objects and called renew on them and measured via
{{monotonicTime}}.
Daryn: please correct me if I am wrong.
bq. Are you aware of JMH?
I wasn't aware until I read the comment as did web search.
But it is very simple to understand that priority queue is not a good data
structure if you want to remove any object other than the top one.
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120700#comment-16120700
]
Kihwal Lee commented on HDFS-12278:
---
+1 looks good.
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120686#comment-16120686
]
Ravi Prakash commented on HDFS-12278:
-
Hi Rushabh! Slightly on a tangent, but how did you benchmark TreeSet and
PriorityQueue? Are you aware of JMH?
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12000) Ozone: Container : Add key versioning support-1
[ https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12000: -- Attachment: HDFS-12000-HDFS-7240.003.patch > Ozone: Container : Add key versioning support-1 > --- > > Key: HDFS-12000 > URL: https://issues.apache.org/jira/browse/HDFS-12000 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Chen Liang > Attachments: HDFS-12000-HDFS-7240.001.patch, > HDFS-12000-HDFS-7240.002.patch, HDFS-12000-HDFS-7240.003.patch, > OzoneVersion.001.pdf > > > The rest interface of ozone supports versioning of keys. This support comes > from the containers and how chunks are managed to support this feature. This > JIRA tracks that feature. Will post a detailed design doc so that we can talk > about this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120650#comment-16120650 ] Ajay Yadav commented on HDFS-12281: --- [~anu],[~xyao] plz review the patch. > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > Attachments: HDFS-12281-HDFS-7240.01.patch > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Status: Patch Available (was: Open) > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > Attachments: HDFS-12281-HDFS-7240.01.patch > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Description: #XML Property: ozone.handler.type Value - local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed # XML Property: ozone.scm.handler.count.key - 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 # XML Property: ozone.metastore.impl- RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB was: # XML Property: ozone.handler.type Value:local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT: distributed # XML Property: ozone.scm.handler.count.key : 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 # XML Property: ozone.metastore.impl: RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > > #XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Attachment: HDFS-12281-HDFS-7240.01.patch > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > Attachments: HDFS-12281-HDFS-7240.01.patch > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Description: # XML Property: ozone.handler.type Value - local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed # XML Property: ozone.scm.handler.count.key - 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 # XML Property: ozone.metastore.impl- RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB was: #XML Property: ozone.handler.type Value - local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed # XML Property: ozone.scm.handler.count.key - 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 # XML Property: ozone.metastore.impl- RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Description: # XML Property: ozone.handler.type Value:local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT: distributed # XML Property: ozone.scm.handler.count.key : 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 # XML Property: ozone.metastore.impl: RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB was: # XML Property: ozone.handler.type Value:local #* Config Name: OZONE_HANDLER_TYPE_DEFAULT: distributed # XML Property: ozone.scm.handler.count.key : 20 #* Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 # XML Property: ozone.metastore.impl: RocksDB #* Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > > # XML Property: ozone.handler.type Value:local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT: distributed > # XML Property: ozone.scm.handler.count.key : 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 > # XML Property: ozone.metastore.impl: RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Description: # XML Property: ozone.handler.type Value:local #* Config Name: OZONE_HANDLER_TYPE_DEFAULT: distributed # XML Property: ozone.scm.handler.count.key : 20 #* Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 # XML Property: ozone.metastore.impl: RocksDB #* Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB was: # XML Property: ozone.handler.type Value:local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT ## Config Value: distributed # XML Property: ozone.scm.handler.count.key : 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 # XML Property: ozone.metastore.impl: RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > > # XML Property: ozone.handler.type Value:local > #* Config Name: OZONE_HANDLER_TYPE_DEFAULT: distributed > # XML Property: ozone.scm.handler.count.key : 20 > #* Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 > # XML Property: ozone.metastore.impl: RocksDB > #* Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Description: # XML Property: ozone.handler.type Value:local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT ## Config Value: distributed # XML Property: ozone.scm.handler.count.key : 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 # XML Property: ozone.metastore.impl: RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > > # XML Property: ozone.handler.type Value:local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT > ## Config Value: distributed > # XML Property: ozone.scm.handler.count.key : 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT: 10 > # XML Property: ozone.metastore.impl: RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT: LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12278:
--
Attachment: HDFS-12278-branch-2.8.001.patch
Attaching a branch-2.8 patch.
Imports conflict with trunk patch.
trunk patch applies nicely to branch-2.
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278-branch-2.8.001.patch, HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120620#comment-16120620
]
Rushabh S Shah commented on HDFS-12278:
---
{{TestDFSStripedOutputStreamWithFailure150}} and
{{TestDFSStripedOutputStreamWithFailure080}} are well known flaky tests.
Both have failed number of times. I ran 4 times each test. Fails with a
probability of 50%
{{TestUnderReplicatedBlocks#testSetRepIncWithUnderReplicatedBlocks}} is timing
out. Tracked via HDFS-9243.
There are no new tests since all the existing test covers the correctness.
Will attach a branch-2.8 patch soon.
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120541#comment-16120541
]
Hadoop QA commented on HDFS-5040:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 277 unchanged - 8 fixed = 281 total (was 285) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-5040 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881054/HDFS-5040.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux dda31a644add 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ec69414 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20622/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20622/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20622/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20622/testReport/ |
| modules | C: hadoop-hdfs-project/h
[jira] [Commented] (HDFS-12000) Ozone: Container : Add key versioning support-1
[
https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120528#comment-16120528
]
Hadoop QA commented on HDFS-12000:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
48s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 9
unchanged - 0 fixed = 10 total (was 9) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 44s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.cblock.TestBufferManager |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
| | org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12000 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881051/HDFS-12000-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 7d93b01e9448 3.13.0
[jira] [Updated] (HDFS-12278) LeaseManager operations are inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12278:
--
Summary: LeaseManager operations are inefficient in 2.8. (was:
LeaseManager#removeLease operation is inefficient in 2.8.)
> LeaseManager operations are inefficient in 2.8.
> ---
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager#removeLease operation is inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120518#comment-16120518
]
Hadoop QA commented on HDFS-12278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881050/HDFS-12278.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7231b045032c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 63cfcb9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20620/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20620/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20620/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20620/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatical
[jira] [Commented] (HDFS-7240) Object store in HDFS
[
https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120502#comment-16120502
]
Elek, Marton commented on HDFS-7240:
Yeah, but it seems for the registration I need an {{@apache.org}} email
address. And no information about invites anywhere.
> Object store in HDFS
>
>
> Key: HDFS-7240
> URL: https://issues.apache.org/jira/browse/HDFS-7240
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf,
> ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS.
> As part of the federation work (HDFS-1052) we separated block storage as a
> generic storage layer. Using the Block Pool abstraction, new kinds of
> namespaces can be built on top of the storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode
> storage, but independent of namespace metadata.
> I will soon update with a detailed design document.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120438#comment-16120438 ] Anu Engineer commented on HDFS-7240: [~elek] Nope, all are welcome. > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, > ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120436#comment-16120436 ] Elek, Marton commented on HDFS-7240: Is this chat for commiters only? > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, > ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager#removeLease operation is inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120418#comment-16120418
]
Daryn Sharp commented on HDFS-12278:
I benched the remove/update/add between the priority queue and original tree
set to simulate renewals.
* 1k files = no difference
* 10k = pq is 3X slower
* 100k = pq is 13X slower
* 200k = pq is 22X slower
> LeaseManager#removeLease operation is inefficient in 2.8.
> -
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12238) Ozone: Add valid trace ID check in sendCommandAsync
[
https://issues.apache.org/jira/browse/HDFS-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120410#comment-16120410
]
Anu Engineer commented on HDFS-12238:
-
Thanks [~vagarychen] for the comments.
bq. java.lang.IllegalStateException: Duplicate trace ID.
This happens due to 2 conditions.
# Someone is reusing the traceID and calling into the sendCommand.
# A retry operation of the command which is not completed is being attempted.
I would guess it is the first case, so it is something that should be fixed in
TestCBlocReadWrite. [~msingh] has a JIRA and probably a patch for that.
bq. java.lang.IllegalArgumentException: Invalid trace ID
This is due to the change the [~ajayydv] did, that is something that we should
look closer.
bq. , if it finds out request ID is missing, set a random one instead of
throwing exception...
I don't think we should do it. If we expect the user app to behave in a certain
way, it should behave in a certain way. We should fix the user app rather than
work around in the library. Also we have tests that try to send in invalid id
and expect an exception, then they will start failing.
> Ozone: Add valid trace ID check in sendCommandAsync
> ---
>
> Key: HDFS-12238
> URL: https://issues.apache.org/jira/browse/HDFS-12238
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Ajay Yadav
> Labels: newbie
> Attachments: HDFS-12238-HDFS-7240.01.patch
>
>
> In the function {{XceiverClientHandler#sendCommandAsync}} we should add a
> check
> {code}
> if(StringUtils.isEmpty(request.getTraceID())) {
> throw new IllegalArgumentException("Invalid trace ID");
> }
> {code}
> To ensure that ozone clients always send a valid trace ID. However, when you
> do that a set of current tests that do add a valid trace ID will fail. So we
> need to fix these tests too.
> {code}
> TestContainerMetrics.testContainerMetrics
> TestOzoneContainer.testBothGetandPutSmallFile
> TestOzoneContainer.testCloseContainer
> TestOzoneContainer.testOzoneContainerViaDataNode
> TestOzoneContainer.testXcieverClientAsync
> TestOzoneContainer.testCreateOzoneContainer
> TestOzoneContainer.testDeleteContainer
> TestContainerServer.testClientServer
> TestContainerServer.testClientServerWithContainerDispatcher
> TestKeys.testPutAndGetKeyWithDnRestart
> {code}
> This is based on a comment from [~vagarychen] in HDFS-11580.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12273) Federation UI
[ https://issues.apache.org/jira/browse/HDFS-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120389#comment-16120389 ] Ravi Prakash commented on HDFS-12273: - I can try to take a look but honestly I'm sorry it won't have very high priority. > Federation UI > - > > Key: HDFS-12273 > URL: https://issues.apache.org/jira/browse/HDFS-12273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: HDFS-12273-HDFS-10467-000.patch, > HDFS-12273-HDFS-10467-001.patch > > > Add the Web UI to the Router to expose the status of the federated cluster. > It includes the federation metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated HDFS-5040: -- Attachment: HDFS-5040.006.patch Fixing minor checkstyle warnings. Findbug warnings are unrelated and so are the test failures. > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.006.patch, HDFS-5040.patch, HDFS-5040.patch, > HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12238) Ozone: Add valid trace ID check in sendCommandAsync
[
https://issues.apache.org/jira/browse/HDFS-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120369#comment-16120369
]
Chen Liang commented on HDFS-12238:
---
Actually, I guess it is probably because some places in the code are not even
setting requestID field...maybe we should either find out all such places and
make them set request ID, OR in {{sendCommandAsync}}, if it finds out request
ID is missing, set a random one instead of throwing exception...
> Ozone: Add valid trace ID check in sendCommandAsync
> ---
>
> Key: HDFS-12238
> URL: https://issues.apache.org/jira/browse/HDFS-12238
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Ajay Yadav
> Labels: newbie
> Attachments: HDFS-12238-HDFS-7240.01.patch
>
>
> In the function {{XceiverClientHandler#sendCommandAsync}} we should add a
> check
> {code}
> if(StringUtils.isEmpty(request.getTraceID())) {
> throw new IllegalArgumentException("Invalid trace ID");
> }
> {code}
> To ensure that ozone clients always send a valid trace ID. However, when you
> do that a set of current tests that do add a valid trace ID will fail. So we
> need to fix these tests too.
> {code}
> TestContainerMetrics.testContainerMetrics
> TestOzoneContainer.testBothGetandPutSmallFile
> TestOzoneContainer.testCloseContainer
> TestOzoneContainer.testOzoneContainerViaDataNode
> TestOzoneContainer.testXcieverClientAsync
> TestOzoneContainer.testCreateOzoneContainer
> TestOzoneContainer.testDeleteContainer
> TestContainerServer.testClientServer
> TestContainerServer.testClientServerWithContainerDispatcher
> TestKeys.testPutAndGetKeyWithDnRestart
> {code}
> This is based on a comment from [~vagarychen] in HDFS-11580.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12238) Ozone: Add valid trace ID check in sendCommandAsync
[
https://issues.apache.org/jira/browse/HDFS-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120364#comment-16120364
]
Chen Liang commented on HDFS-12238:
---
Hi [~ajayydv], I only checked {{TestBufferManager}} and {{TestCBlockReadWrite}}
because they are currently failing consistently due to
{code}
java.lang.IllegalStateException: Duplicate trace ID. Command with this trace ID
is already executing. Please ensure that trace IDs are not reused. ID:
at
org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139)
{code}
So I thought this JIRA might solve this, but seems with the patch they are
still failing, but with a different error
{code}
java.lang.IllegalArgumentException: Invalid trace ID
at
org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:131)
{code}
Could you please check on this?
> Ozone: Add valid trace ID check in sendCommandAsync
> ---
>
> Key: HDFS-12238
> URL: https://issues.apache.org/jira/browse/HDFS-12238
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Ajay Yadav
> Labels: newbie
> Attachments: HDFS-12238-HDFS-7240.01.patch
>
>
> In the function {{XceiverClientHandler#sendCommandAsync}} we should add a
> check
> {code}
> if(StringUtils.isEmpty(request.getTraceID())) {
> throw new IllegalArgumentException("Invalid trace ID");
> }
> {code}
> To ensure that ozone clients always send a valid trace ID. However, when you
> do that a set of current tests that do add a valid trace ID will fail. So we
> need to fix these tests too.
> {code}
> TestContainerMetrics.testContainerMetrics
> TestOzoneContainer.testBothGetandPutSmallFile
> TestOzoneContainer.testCloseContainer
> TestOzoneContainer.testOzoneContainerViaDataNode
> TestOzoneContainer.testXcieverClientAsync
> TestOzoneContainer.testCreateOzoneContainer
> TestOzoneContainer.testDeleteContainer
> TestContainerServer.testClientServer
> TestContainerServer.testClientServerWithContainerDispatcher
> TestKeys.testPutAndGetKeyWithDnRestart
> {code}
> This is based on a comment from [~vagarychen] in HDFS-11580.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12284) Router support for Kerberos authentication and delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-12284: - Summary: Router support for Kerberos authentication and delegation tokens (was: Router support for Kerberos and delegation tokens) > Router support for Kerberos authentication and delegation tokens > > > Key: HDFS-12284 > URL: https://issues.apache.org/jira/browse/HDFS-12284 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: security >Reporter: Zhe Zhang >Assignee: xiangguang zheng > Fix For: HDFS-10467 > > > HDFS Router should support Kerberos authentication and issuing / managing > HDFS delegation tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12284) Router support for Kerberos and delegation tokens
Zhe Zhang created HDFS-12284: Summary: Router support for Kerberos and delegation tokens Key: HDFS-12284 URL: https://issues.apache.org/jira/browse/HDFS-12284 Project: Hadoop HDFS Issue Type: Sub-task Components: security Reporter: Zhe Zhang Assignee: xiangguang zheng HDFS Router should support Kerberos authentication and issuing / managing HDFS delegation tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12000) Ozone: Container : Add key versioning support-1
[
https://issues.apache.org/jira/browse/HDFS-12000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12000:
--
Attachment: HDFS-12000-HDFS-7240.002.patch
Post v002 patch to fix findbugs, checkstyle and license issues. The failed
tests are unrelated. All passed except {{TestBufferManager}} and
{{TestCBlockReadWrite}}, seems these two are failing consistently recently due
to
{code}
java.lang.IllegalStateException: Duplicate trace ID. Command with this trace ID
is already executing. Please ensure that trace IDs are not reused. ID:
at
org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139)
{code}
Also {{TestKeysRatis}} and {{TestKeys}} are known failing tests.
> Ozone: Container : Add key versioning support-1
> ---
>
> Key: HDFS-12000
> URL: https://issues.apache.org/jira/browse/HDFS-12000
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Chen Liang
> Attachments: HDFS-12000-HDFS-7240.001.patch,
> HDFS-12000-HDFS-7240.002.patch, OzoneVersion.001.pdf
>
>
> The rest interface of ozone supports versioning of keys. This support comes
> from the containers and how chunks are managed to support this feature. This
> JIRA tracks that feature. Will post a detailed design doc so that we can talk
> about this feature.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120336#comment-16120336
]
Hadoop QA commented on HDFS-5040:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 64 new + 277 unchanged - 8 fixed = 341 total (was 285) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-5040 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881040/HDFS-5040.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5cc61baa07d6 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 63cfcb9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20618/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20618/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20618/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20618/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Buil
[jira] [Updated] (HDFS-12278) LeaseManager#removeLease operation is inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12278:
--
Status: Patch Available (was: Open)
> LeaseManager#removeLease operation is inefficient in 2.8.
> -
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12278) LeaseManager#removeLease operation is inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah updated HDFS-12278:
--
Attachment: HDFS-12278.patch
Very small change.
Switched back to TreeSet from Priority Queue.
Ran basic LeaseManager related test cases.
> LeaseManager#removeLease operation is inefficient in 2.8.
> -
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-12278.patch
>
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12254) Upgrade JUnit from 4 to 5 in hadoop-hdfs
[ https://issues.apache.org/jira/browse/HDFS-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120316#comment-16120316 ] Ajay Yadav commented on HDFS-12254: --- hi [~ajisakaa] , As i understand , we can do it in two ways. # 1. Update the junit dependency in hadoop-main to junit5 with junit-jupiter-engine. which will require changes in most of the test cases to move them to new junit5 api and platform. (More riskier) # 2. Update the junit dependency in hadoop-main to junit5 while maintaining backward compatibility to test cases built in junit4 using (junit-vintage-engine). As a next step we can create new test cases using junit 5 api and move old test cases to junit5 in steps. This will be incremental change with less risk to breaking old test cases. Any ideas, suggestions on this? > Upgrade JUnit from 4 to 5 in hadoop-hdfs > > > Key: HDFS-12254 > URL: https://issues.apache.org/jira/browse/HDFS-12254 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Akira Ajisaka >Assignee: Ajay Yadav > > Feel free to create sub-tasks for each module. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12202) Provide new set of FileSystem API to bypass external attribute provider
[
https://issues.apache.org/jira/browse/HDFS-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120313#comment-16120313
]
Yongjun Zhang commented on HDFS-12202:
--
Thanks for the feedback [~asuresh] [~manojg] and [~chris.douglas]. Sorry for my
delayed reply since I was out for some time.
[~asuresh]:
{quote}
but I feel we should maybe explore ways around having to modify the HDFS API by
configuring the External provider to return the underlying Attributes (and
possibly bypass permission checks) for just a white-listed set of users (and/or
a configured set of name-spaces) - this implies that performing distcp (without
copying over the externally over-laid attributes) might be restricted to only a
few users of the cluster - but from a practical standpoint, I think it should
be reasonable, since I believe that for most clusters, this cluster-to-cluster
copying does not happen very often and I usually performed by an cluster admin
/ manager.
{quote}
This is an interesting idea! I wish there is a solution that we can avoid
modifying HDFS API. For a white-listed set of users, however, I would assume
that some of the calls issued by these users need to access the external
attributes, and some don't, we don't have a way to distinguish. For example,
the same user may run different applications too than distcp too. How we are
going to solve this with the white-list solution?
In addition, ANY user can run distcp, and distcp can happen within a same
cluster too. If we want to these, would it be too restrictive. Thoughts?
[~manojg]:
{quote}
HDFS-12203 on the similar lines for requesting the external provider to run in
pass through mode
{quote}
we need to know when to pass through. I think in your case, we are talking
about letting snapshotDiff to pass through, which is feasible because
snapshotDiff is at NN side and it has access to external attribute API. But for
distcp issue we are talking about here, we have the problem of not knowing when
to pass through, because only users knows when to pass through and we don't
have a way to fill the gap between user (accessing FileSystem API only) and the
pass through API of external attribute provider. The solution proposed in this
jira is one possible way to fill the gap.
[~chris.douglas]:
{quote}
This is a pretty narrow use case.
{quote}
Agree. Only when external attribute provider is enabled, and in the context of
distcp.
{quote}
Extending FileSystem is a very hard sell, since it would also add this flag to
the protocol.
{quote}
Agree the change is wide, and I wish there is a simpler way.
{quote}
Not only would this approach not work for old clusters, it would silently
return the unfiltered results.
{quote}
Agree that any solution here would require a change on the old cluster.
{quote}
Moreover, every FileSystem other than HDFS wouldn't support this. Are there
other use cases?
{quote}
Right now distcp with external attribute provider enabled is the only use case
I can see. Since distcp is a client that only access FileSystem API, I was
proposing extending the API. My thinking was, for FileSystems that don't care
about this, the bypass parameter is simply ignored.
Thanks again, and hope to hear your further thoughts...
.
> Provide new set of FileSystem API to bypass external attribute provider
> ---
>
> Key: HDFS-12202
> URL: https://issues.apache.org/jira/browse/HDFS-12202
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs, hdfs-client
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> HDFS client uses
> {code}
> /**
>* Return a file status object that represents the path.
>* @param f The path we want information from
>* @return a FileStatus object
>* @throws FileNotFoundException when the path does not exist
>* @throws IOException see specific implementation
>*/
> public abstract FileStatus getFileStatus(Path f) throws IOException;
> /**
>* List the statuses of the files/directories in the given path if the path
> is
>* a directory.
>*
>* Does not guarantee to return the List of files/directories status in a
>* sorted order.
>*
>* Will not return null. Expect IOException upon access error.
>* @param f given path
>* @return the statuses of the files/directories in the given patch
>* @throws FileNotFoundException when the path does not exist
>* @throws IOException see specific implementation
>*/
> public abstract FileStatus[] listStatus(Path f) throws
> FileNotFoundException,
> IOException;
> {code}
> to get FileStatus of files.
> When external attribute provider (INodeAttributeProvider) is enabled for a
> cluster, the externa
[jira] [Commented] (HDFS-12221) Replace xcerces in XmlEditsVisitor
[ https://issues.apache.org/jira/browse/HDFS-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120268#comment-16120268 ] Ajay Yadav commented on HDFS-12221: --- [~andrew.wang], [~eddyxu] Added updated binary file to resolve the HDFS test which is failing. (i.e TestOfflineEditsViewer#testStored). Please test the patch with updated binary file "editsStored" > Replace xcerces in XmlEditsVisitor > --- > > Key: HDFS-12221 > URL: https://issues.apache.org/jira/browse/HDFS-12221 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha4 >Reporter: Lei (Eddy) Xu >Assignee: Ajay Yadav > Attachments: editsStored, fsimage_hdfs-12221.xml, > HDFS-12221.01.patch, HDFS-12221.02.patch, HDFS-12221.03.patch, > HDFS-12221.04.patch, HDFS-12221.05.patch > > > XmlEditsVisitor should use new XML capability in the newer JDK, to make JAR > shading easier (HADOOP-14672) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12221) Replace xcerces in XmlEditsVisitor
[ https://issues.apache.org/jira/browse/HDFS-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12221: -- Attachment: editsStored HDFS-12221.05.patch > Replace xcerces in XmlEditsVisitor > --- > > Key: HDFS-12221 > URL: https://issues.apache.org/jira/browse/HDFS-12221 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha4 >Reporter: Lei (Eddy) Xu >Assignee: Ajay Yadav > Attachments: editsStored, fsimage_hdfs-12221.xml, > HDFS-12221.01.patch, HDFS-12221.02.patch, HDFS-12221.03.patch, > HDFS-12221.04.patch, HDFS-12221.05.patch > > > XmlEditsVisitor should use new XML capability in the newer JDK, to make JAR > shading easier (HADOOP-14672) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12266) Ozone : add debug cli to hdfs script
[ https://issues.apache.org/jira/browse/HDFS-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120255#comment-16120255 ] Chen Liang commented on HDFS-12266: --- Thanks [~elek] for the catch! I totally missed it...done what you suggested by reverting HDFS-11836... > Ozone : add debug cli to hdfs script > > > Key: HDFS-12266 > URL: https://issues.apache.org/jira/browse/HDFS-12266 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Minor > Fix For: HDFS-7240 > > Attachments: HDFS-12266-HDFS-7240.001.patch > > > The debug CLI (which converts metadata levelDB/RocksDB file to sqlite file) > is still missing in hdfs script, this JIRA adds it as one of the hdfs > subcommands. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120243#comment-16120243
]
Yongjun Zhang commented on HDFS-12217:
--
Hi [~manojg] and [~jojochuang],
Thanks for continuing to drive on this when I was out. I am glad to see that we
have converged on the solution.
> HDFS snapshots doesn't capture all open files when one of the open files is
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch,
> HDFS-12217.03.patch, HDFS-12217.04.patch, HDFS-12217.05.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the
> open files. Just like all other files, these open files in the snapshots will
> remain immutable. But, sometimes it is found that snapshots fail to capture
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode
> corresponding to an active lease
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few
> leases there by causing the caller to trip over and not return all the open
> files needed by the snapshot manager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120227#comment-16120227 ] Anoop Sam John commented on HDFS-10285: --- HBase can get benefited from this feature. The scenario is as below HBase allow the WAL files to be kept in low latency devices using the HSM feature. (ALL_SSD/ ONE_SSD etc) There is a directory for keeping all active WALs and we config the policy for that. After certain time, the WAL file will become inactive as all the data in that is eventually getting flushed into HFiles. We will then archive it. There is an archive directory and the archive op is done via a rename to a file under the archive dir. Obviously the archive dir won't have any policy configured. By default we will keep the WAL files under archive dir for some more min and then delete them. If the WAL can get deleted it is fine even if the blocks of the WAL files continue to be in low latency device. But there are some features and scenarios under which the deletion of WAL from archive can get delayed. Few eg:s Cross cluster replication in place and the peer replica is slow/down. HBase do inter cluster replication by reading the WAL. As long as the WAL cells are read and passed to other cluster, we can not delete Backup feature in use and the backup refers to WAL files (Snapshot feature also) Incremental backup is enabled. Unless an incremental backup is taken, WALs in that time range can not be deleted. Same for HFiles. After compaction, the compacted away files are archived and if they are referred by some active snapshots, we may not be able to delete them immediately. So it makes all sense to make use of this feature for moving the File blocks out of low latency devices so as to free space in it. Once this feature is GA in a version and we can open up jira to make use of it. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12278) LeaseManager#removeLease operation is inefficient in 2.8.
[
https://issues.apache.org/jira/browse/HDFS-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120217#comment-16120217
]
Daryn Sharp commented on HDFS-12278:
For context regarding the impact of the change to a priority queue: Hours
after a 2.8 upgrade, avg rpc processing time increased from sub-ms to 21ms.
Rpc queue time was multiple seconds. Killing large jobs only made it worse.
The fair call queue was completely overflowing for ~5h. I haven't seen
anything this horrific in many years.
While the NN log was spewing logs of skipping calls from timing out clients, we
noticed lease monitor recovery log messages ~5-12ms apart during which time the
lease monitor holds the write lock. Killing jobs made it worse because it
created more orphaned leases.
> LeaseManager#removeLease operation is inefficient in 2.8.
> -
>
> Key: HDFS-12278
> URL: https://issues.apache.org/jira/browse/HDFS-12278
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
>
> After HDFS-6757, LeaseManager #removeLease became expensive.
> HDFS-6757 changed the {{sortedLeases}} object from TreeSet to PriorityQueue.
> Previously the {{remove(Object)}} operation from {{sortedLeases}} was {{O(log
> n)}} but after the change it became {{O( n)}} since it has to find the object
> first.
> Recently we had an incident in one of our production cluster just hours after
> we upgraded from 2.7 to 2.8
> The {{sortledLeases}} object had approximately 100,000 items within it.
> While removing the lease, it will acquire the LeaseManager lock and that will
> slow down the lookup of lease also.
> HDFS-6757 is a good improvement which replaced the path by inode id.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated HDFS-5040: -- Attachment: HDFS-5040.005.patch bq. Add fail assertion for exception cases. you can use ContractTestUtils.fail If I understood that correctly, I have added try-catch for allowed=true cases to fail if any exception is encountered. I have used the Assert.fail() call as the previous patch. Fixed the test failures in TestAuditLogger, TestDFSAdminWithHA and TestNameNodeMXBean. Other failures are unrelated and flaky. Will wait on precommit before asking for further review. Thanks [~brahmareddy] for the reviews so far. > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.004.patch, > HDFS-5040.005.patch, HDFS-5040.patch, HDFS-5040.patch, HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120112#comment-16120112
]
Hadoop QA commented on HDFS-12274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
24s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 1s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.cblock.TestBufferManager |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881022/HDFS-12274-HDFS-7240.002.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux db59fcfc6456 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 43d3811 |
| Default Java | 1.8.0_131 |
| shellcheck | v0.4.6 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20617/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20617/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-pro
[jira] [Updated] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-12157: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.7.5 2.8.2 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2, branch-2.8, branch-2.8.2 and branch-2.7. Thanks for working on this Vinay and thanks for reviewing it Brahma. > Do fsyncDirectory(..) outside of FSDataset lock > --- > > Key: HDFS-12157 > URL: https://issues.apache.org/jira/browse/HDFS-12157 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Fix For: 2.9.0, 3.0.0-beta1, 2.8.2, 2.7.5 > > Attachments: HDFS-12157-01.patch, HDFS-12157-branch-2-01.patch, > HDFS-12157-branch-2.7-01.patch, HDFS-12157-branch-2.7-01.patch > > > HDFS-5042 introduced fsyncDirectory(..) to save blocks from power failure. > Do it outside of FSDataset lock to avoid overall performance degradation if > disk takes more time to sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119984#comment-16119984 ] Hudson commented on HDFS-12157: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12153 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12153/]) HDFS-12157. Do fsyncDirectory(..) outside of FSDataset lock. Contributed (kihwal: rev 69afa26f19adad4c630a307c274130eb8b697141) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java > Do fsyncDirectory(..) outside of FSDataset lock > --- > > Key: HDFS-12157 > URL: https://issues.apache.org/jira/browse/HDFS-12157 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HDFS-12157-01.patch, HDFS-12157-branch-2-01.patch, > HDFS-12157-branch-2.7-01.patch, HDFS-12157-branch-2.7-01.patch > > > HDFS-5042 introduced fsyncDirectory(..) to save blocks from power failure. > Do it outside of FSDataset lock to avoid overall performance degradation if > disk takes more time to sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12268) Ozone: Add metrics for pending storage container requests
[
https://issues.apache.org/jira/browse/HDFS-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119941#comment-16119941
]
Hadoop QA commented on HDFS-12268:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 47s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.ozone.container.transport.server.TestContainerServer |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.ksm.TestMultipleContainerReadWrite |
| | hadoop.ozone.TestContainerOperations |
| | hadoop.cblock.TestBufferManager |
| | hadoop.ozone.container.metrics.TestContainerMetrics |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.ozone.scm.TestXceiverClientManager |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.cblock.TestLocalBlockCache |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.scm.TestContainerSmallFile |
| | hadoop.ozone.ksm.TestKSMSQLCli |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.hdfs.
[jira] [Updated] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12274:
--
Attachment: HDFS-12274-HDFS-7240.002.patch
Patch v002 with checkstyle fix.
> Ozone: Corona: move corona from test to tools package
> -
>
> Key: HDFS-12274
> URL: https://issues.apache.org/jira/browse/HDFS-12274
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12274-HDFS-7240.000.patch,
> HDFS-12274-HDFS-7240.001.patch, HDFS-12274-HDFS-7240.002.patch
>
>
> This jira is to move {{Corona}} from test to tools package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119932#comment-16119932 ] Kihwal Lee commented on HDFS-12157: --- +1 LGTM. > Do fsyncDirectory(..) outside of FSDataset lock > --- > > Key: HDFS-12157 > URL: https://issues.apache.org/jira/browse/HDFS-12157 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HDFS-12157-01.patch, HDFS-12157-branch-2-01.patch, > HDFS-12157-branch-2.7-01.patch, HDFS-12157-branch-2.7-01.patch > > > HDFS-5042 introduced fsyncDirectory(..) to save blocks from power failure. > Do it outside of FSDataset lock to avoid overall performance degradation if > disk takes more time to sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119912#comment-16119912
]
Hadoop QA commented on HDFS-12274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
6s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
25s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881002/HDFS-12274-HDFS-7240.001.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux 69ddf0a72a14 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 43d3811 |
| Default Java | 1.8.0_131 |
| shellcheck | v0.4.6 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20615/art
[jira] [Commented] (HDFS-12117) HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface
[ https://issues.apache.org/jira/browse/HDFS-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119801#comment-16119801 ] Hudson commented on HDFS-12117: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12151 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12151/]) HDFS-12117. HttpFS does not seem to support SNAPSHOT related methods for (weichiu: rev 8a4bff02c1534c6bf529726f2bbe414ac4c172e8) * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSParametersProvider.java * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/server/TestHttpFSServer.java * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java > HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST > Interface > --- > > Key: HDFS-12117 > URL: https://issues.apache.org/jira/browse/HDFS-12117 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 3.0.0-alpha3 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12117.003.patch, HDFS-12117.004.patch, > HDFS-12117.005.patch, HDFS-12117.006.patch, HDFS-12117.patch.01, > HDFS-12117.patch.02 > > > Currently, HttpFS is lacking implementation for SNAPSHOT related methods from > WebHDFS REST interface as defined by WebHDFS documentation [WebHDFS > documentation|https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Snapshot_Operations] > I would like to work on this implementation, following the existing design > approach already implemented by other WebHDFS methods on current HttpFS > project, so I'll be proposing an initial patch soon for reviews. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6489) DFS Used space is not correct computed on frequent append operations
[
https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119797#comment-16119797
]
Brahma Reddy Battula commented on HDFS-6489:
bq.Even with this, dfsUsed and numblocks counting is all messed up. e.g.
FsDatasetImpl.removeOldBlock calls decDfsUsedAndNumBlocks twice (so even though
dfsUsed is correctly decremented, numBlocks is not)
Nope. I believe you read this wrong.
{{FsDatasetImpl#removeOldReplica}} call two seperate calls
{{onBlockFileDeletion(..)}} and {{onMetaFileDeletion(...)}} for {{blockfile}}
and {{metafile}} respectively.
{code}
// Remove the old replicas
if (replicaInfo.deleteBlockData() || !replicaInfo.blockDataExists()) {
FsVolumeImpl volume = (FsVolumeImpl) replicaInfo.getVolume();
volume.onBlockFileDeletion(bpid, replicaInfo.getBytesOnDisk());
if (replicaInfo.deleteMetadata() || !replicaInfo.metadataExists()) {
volume.onMetaFileDeletion(bpid, replicaInfo.getMetadataLength());
}
{code}
*Code from {{FsVolumeImpl.java}}*
{code}
void onBlockFileDeletion(String bpid, long value) {
decDfsUsedAndNumBlocks(bpid, value, true);
if (isTransientStorage()) {
dataset.releaseLockedMemory(value, true);
}
}
void onMetaFileDeletion(String bpid, long value) {
decDfsUsedAndNumBlocks(bpid, value, false);
}
private void decDfsUsedAndNumBlocks(String bpid, long value,
boolean blockFileDeleted) {
try(AutoCloseableLock lock = dataset.acquireDatasetLock()) {
BlockPoolSlice bp = bpSlices.get(bpid);
if (bp != null) {
bp.decDfsUsed(value);
if (blockFileDeleted) {
bp.decrNumBlocks();
}
}
}
}
{code}
{{onBlockFileDeletion(..)}} calls {{decDfsUsedAndNumBlocks(bpid, value,
true);}} with {{blockFileDeleted}} flag as {{true}} to drement the
{{numblocks}},where as {{onMetaFileDeletion(...)}} calls
{{decDfsUsedAndNumBlocks(bpid, value, false);}} with {{blockFileDeleted}} flag
as {{false}}.Because,no need to decrement the {{numblocks}} for metafile
deletion.
bq.Also, what do you think about a robust unit-test framework to find out all
these issues?
Only way is to list all write/delete cases and write tests for that
*Comments for this Jira*
1) only {{incDfsUsed()}} should be used, as {{numBlocks}} will be updated
during {{createRbw()}} for new blocks. For {{append}} incrementing
{{numBlocks}} not required.
2) Previous metadata length also should be deducted.
{code}
860 if(b instanceof ReplicaInfo) {
861 ReplicaInfo replicaInfo = ((ReplicaInfo) b);
862 if(replicaInfo.getState() == ReplicaState.RBW) {
863 ReplicaInPipeline rip = (ReplicaInPipeline) replicaInfo;
864 // rip.getOriginalBytesReserved() - rip.getBytesReserved()
865 // is the amount of data that was written to the replica
866 long bytesAdded = rip.getOriginalBytesReserved() -
867 rip.getBytesReserved() + replicaInfo.getMetaFile().length();
868 incDfsUsedAndNumBlocks(bpid, bytesAdded);
869 }
870 }
{code}
Sorry for late reply.
> DFS Used space is not correct computed on frequent append operations
>
>
> Key: HDFS-6489
> URL: https://issues.apache.org/jira/browse/HDFS-6489
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.2.0, 2.7.1, 2.7.2
>Reporter: stanley shi
>Assignee: Weiwei Yang
> Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch,
> HDFS-6489.003.patch, HDFS-6489.004.patch, HDFS-6489.005.patch,
> HDFS-6489.006.patch, HDFS-6489.007.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space
> on each block write operation. This is correct in most scenario (create new
> file), but sometimes it will behave in-correct(append small data to a large
> block).
> For example, I have a file with only one block(say, 60M). Then I try to
> append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the
> block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large
> number of files (1000+), assume the block size is 32M (half of the default
> value), then the dfs used will be increased 1000*32M = 32G on each append to
> the files; but actually I only write 10K bytes; this will cause the datanode
> to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
> BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received
> exception org.apach
[jira] [Commented] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119761#comment-16119761
]
Nandakumar commented on HDFS-12274:
---
Thanks [~cheersyang] for the review. Uploaded patch v001 with findbug fixes.
> Ozone: Corona: move corona from test to tools package
> -
>
> Key: HDFS-12274
> URL: https://issues.apache.org/jira/browse/HDFS-12274
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12274-HDFS-7240.000.patch,
> HDFS-12274-HDFS-7240.001.patch
>
>
> This jira is to move {{Corona}} from test to tools package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12274) Ozone: Corona: move corona from test to tools package
[
https://issues.apache.org/jira/browse/HDFS-12274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandakumar updated HDFS-12274:
--
Attachment: HDFS-12274-HDFS-7240.001.patch
> Ozone: Corona: move corona from test to tools package
> -
>
> Key: HDFS-12274
> URL: https://issues.apache.org/jira/browse/HDFS-12274
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nandakumar
>Assignee: Nandakumar
> Attachments: HDFS-12274-HDFS-7240.000.patch,
> HDFS-12274-HDFS-7240.001.patch
>
>
> This jira is to move {{Corona}} from test to tools package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12214) [SPS]: Fix review comments of StoragePolicySatisfier feature
[ https://issues.apache.org/jira/browse/HDFS-12214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119730#comment-16119730 ] Rakesh R commented on HDFS-12214: - bq. 5) can we update statistics ..? org.apache.hadoop.hdfs.DistributedFileSystem#satisfyStoragePolicy, [~brahmareddy], I've added this point in HDFS-12228 sub-task. > [SPS]: Fix review comments of StoragePolicySatisfier feature > > > Key: HDFS-12214 > URL: https://issues.apache.org/jira/browse/HDFS-12214 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-12214-HDFS-10285-00.patch, > HDFS-12214-HDFS-10285-01.patch, HDFS-12214-HDFS-10285-02.patch, > HDFS-12214-HDFS-10285-03.patch, HDFS-12214-HDFS-10285-04.patch, > HDFS-12214-HDFS-10285-05.patch, HDFS-12214-HDFS-10285-06.patch > > > This sub-task is to address [~andrew.wang]'s review comments. Please refer > the [review > comment|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16103734&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16103734] > in HDFS-10285 umbrella jira. > # Rename configuration property 'dfs.storage.policy.satisfier.activate' to > 'dfs.storage.policy.satisfier.enabled' > # Disable SPS feature by default. > # Rather than using the acronym (which a user might not know), maybe rename > "-isSpsRunning" to "-isSatisfierRunning" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12228) [SPS]: Add storage policy satisfier related metrics
[
https://issues.apache.org/jira/browse/HDFS-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119729#comment-16119729
]
Rakesh R commented on HDFS-12228:
-
Adding one more comment here, this has to be taken care during implementation.
Update statistics for
org.apache.hadoop.hdfs.DistributedFileSystem#satisfyStoragePolicy, [please
refer Brahma's comment
|https://issues.apache.org/jira/browse/HDFS-12214?focusedCommentId=16109006&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16109006].
> [SPS]: Add storage policy satisfier related metrics
> ---
>
> Key: HDFS-12228
> URL: https://issues.apache.org/jira/browse/HDFS-12228
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>
> This jira to discuss and implement metrics needed for SPS feature.
> Below are few metrics:
> # count of {{inprogress}} block movements
> # count of {{successful}} block movements
> # count of {{failed}} block movements
> Need to analyse and add more.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12266) Ozone : add debug cli to hdfs script
[
https://issues.apache.org/jira/browse/HDFS-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119721#comment-16119721
]
Elek, Marton edited comment on HDFS-12266 at 8/9/17 10:59 AM:
--
Oh, I found it, there was a typo in the commit of HDFS-11836 (commnad):
{code}
hadoop_add_subcommnad "sqlconvert" "convert ozone leveldb files into sqlite db
file for debug purpose"
{code}
So {{sqlconvert}} was not visible from the command line.
I suggest to revert the older commit: HDFS-11836
was (Author: elek):
Oh, I found it, there was a typo in the commit of HDFS-11836:
{code}
hadoop_add_subcommnad "sqlconvert" "convert ozone leveldb files into sqlite db
file for debug purpose"
{code}
So {{sqlconvert}} was not visible from the command line.
I suggest to revert the older commit: HDFS-11836
> Ozone : add debug cli to hdfs script
>
>
> Key: HDFS-12266
> URL: https://issues.apache.org/jira/browse/HDFS-12266
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-12266-HDFS-7240.001.patch
>
>
> The debug CLI (which converts metadata levelDB/RocksDB file to sqlite file)
> is still missing in hdfs script, this JIRA adds it as one of the hdfs
> subcommands.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12266) Ozone : add debug cli to hdfs script
[
https://issues.apache.org/jira/browse/HDFS-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119721#comment-16119721
]
Elek, Marton commented on HDFS-12266:
-
Oh, I found it, there was a typo in the commit of HDFS-11836:
{code}
hadoop_add_subcommnad "sqlconvert" "convert ozone leveldb files into sqlite db
file for debug purpose"
{code}
So {{sqlconvert}} was not visible from the command line.
I suggest to revert the older commit: HDFS-11836
> Ozone : add debug cli to hdfs script
>
>
> Key: HDFS-12266
> URL: https://issues.apache.org/jira/browse/HDFS-12266
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Minor
> Fix For: HDFS-7240
>
> Attachments: HDFS-12266-HDFS-7240.001.patch
>
>
> The debug CLI (which converts metadata levelDB/RocksDB file to sqlite file)
> is still missing in hdfs script, this JIRA adds it as one of the hdfs
> subcommands.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12266) Ozone : add debug cli to hdfs script
[ https://issues.apache.org/jira/browse/HDFS-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119696#comment-16119696 ] Elek, Marton commented on HDFS-12266: - I think it is a duplicate of HDFS-11836 and one of them could be reverted. But maybe I missed something. > Ozone : add debug cli to hdfs script > > > Key: HDFS-12266 > URL: https://issues.apache.org/jira/browse/HDFS-12266 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Minor > Fix For: HDFS-7240 > > Attachments: HDFS-12266-HDFS-7240.001.patch > > > The debug CLI (which converts metadata levelDB/RocksDB file to sqlite file) > is still missing in hdfs script, this JIRA adds it as one of the hdfs > subcommands. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12221) Replace xcerces in XmlEditsVisitor
[
https://issues.apache.org/jira/browse/HDFS-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119674#comment-16119674
]
Hadoop QA commented on HDFS-12221:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-project-dist hadoop-yarn-project/hadoop-yarn
hadoop-mapreduce-project/hadoop-mapreduce-client
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
59s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
7s{color} | {color:green} root generated 0 new + 1371 unchanged - 6 fixed =
1371 total (was 1377) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
9s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-project-dist hadoop-yarn-project/hadoop-yarn
hadoop-mapreduce-project/hadoop-mapreduce-client
hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
12s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-project-dist in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 23s{color}
| {color:red} hadoop-yarn
[jira] [Commented] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
[
https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119670#comment-16119670
]
Weiwei Yang commented on HDFS-12283:
+[~anu] and [~xyao] in the loop, feel free to comment please. :)
> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -
>
> Key: HDFS-12283
> URL: https://issues.apache.org/jira/browse/HDFS-12283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container
> blocks which are under deletion. It maintains info about under-deletion
> container blocks that notified by KSM, and the state how it is processed. We
> can use RocksDB to implement the 1st version of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and
> multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode,
> it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
> means the transaction eventually failed after some retries, 5 is the max
> number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with
> RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
[
https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang reassigned HDFS-12283:
--
Assignee: Yuanbo Liu (was: Weiwei Yang)
> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -
>
> Key: HDFS-12283
> URL: https://issues.apache.org/jira/browse/HDFS-12283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Yuanbo Liu
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container
> blocks which are under deletion. It maintains info about under-deletion
> container blocks that notified by KSM, and the state how it is processed. We
> can use RocksDB to implement the 1st version of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and
> multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode,
> it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
> means the transaction eventually failed after some retries, 5 is the max
> number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with
> RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
[
https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119663#comment-16119663
]
Weiwei Yang edited comment on HDFS-12283 at 8/9/17 10:10 AM:
-
Some more thoughts. Each key value pair in the log is
{code}
// A record in the log
{ long txid : BlockDeletionTransaction tx}
// where the value is (protobuf message)
BlockDeletionTransaction {
long txid;
String containerName;
List blockIDs;
}
{code}
Following API needs to be implemented
{code}
// Returns a certain size list of transactions
// criteria: each 0<= processedCount < 5
// We may need to maintain a position while scanning the log
// to avoid repetitively scanning a certain range of records
List getBlockDeletionTransactions(int count);
// Increment processedCount for given list of transactions by 1
// if count > 5, reset to -1
void incrementProcessedCount(List txIDs);
// Delete transactions from the log based on their IDs,
// the certain list of records will be removed from the DB.
void commitTransactions(List txIDs);
{code}
Please let me know if this makes sense.
Thanks
was (Author: cheersyang):
Some more thoughts. Each key value pair in the log is
{code}
// A record in the log
{ long txid : BlockDeletionTransaction tx}
// where the value is (protobuf message)
BlockDeletionTransaction {
long txid;
String containerName;
List blockIDs;
}
{code}
Following API needs to be implemented
{code}
// Returns a certain size list of transactions
// criteria: each 0<= processedCount < 5
// We may need to maintain a position while scanning the log
// to avoid repetitively scanning a certain range of records
List getBlockDeletionTransactions(int count);
// Increment processedCount for given list of transactions by 1
// if count > 5, reset to -1
void incrementProcessedCount(List txIDs);
// Delete transactions from the log based on their IDs
void commitTransactions(List txIDs);
{code}
Please let me know if this makes sense.
Thanks
> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -
>
> Key: HDFS-12283
> URL: https://issues.apache.org/jira/browse/HDFS-12283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container
> blocks which are under deletion. It maintains info about under-deletion
> container blocks that notified by KSM, and the state how it is processed. We
> can use RocksDB to implement the 1st version of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and
> multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode,
> it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
> means the transaction eventually failed after some retries, 5 is the max
> number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with
> RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
[
https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119663#comment-16119663
]
Weiwei Yang edited comment on HDFS-12283 at 8/9/17 10:09 AM:
-
Some more thoughts. Each key value pair in the log is
{code}
// A record in the log
{ long txid : BlockDeletionTransaction tx}
// where the value is (protobuf message)
BlockDeletionTransaction {
long txid;
String containerName;
List blockIDs;
}
{code}
Following API needs to be implemented
{code}
// Returns a certain size list of transactions
// criteria: each 0<= processedCount < 5
// We may need to maintain a position while scanning the log
// to avoid repetitively scanning a certain range of records
List getBlockDeletionTransactions(int count);
// Increment processedCount for given list of transactions by 1
// if count > 5, reset to -1
void incrementProcessedCount(List txIDs);
// Delete transactions from the log based on their IDs
void commitTransactions(List txIDs);
{code}
Please let me know if this makes sense.
Thanks
was (Author: cheersyang):
Some more thoughts. Each key value pair in the log is
{code}
// A record in the log
{ long txid : BlockDeletionTransaction tx}
// where the value is (protobuf message)
BlockDeletionTransaction {
long txid;
String containerName;
List blockIDs;
}
{code}
Following API needs to be implemented
{code}
// Returns a certain size list of transactions
// criteria: each 0<= processedCount < 5
// We may need to maintain a position while scanning the log
// to avoid repetitively scanning a certain range of records
List getBlockDeletionTransactions(int count);
// Increment processedCount for given list of transactions by 1
// if count > 5, reset to -1
void incrementProcessedCount(List txIDs);
// Delete transactions from the log based on their IDs
void commitTransactions(List txIDs);
{code}
Please let me know if this makes sense.
Thanks
> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -
>
> Key: HDFS-12283
> URL: https://issues.apache.org/jira/browse/HDFS-12283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container
> blocks which are under deletion. It maintains info about under-deletion
> container blocks that notified by KSM, and the state how it is processed. We
> can use RocksDB to implement the 1st version of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and
> multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode,
> it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
> means the transaction eventually failed after some retries, 5 is the max
> number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with
> RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
[
https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119663#comment-16119663
]
Weiwei Yang edited comment on HDFS-12283 at 8/9/17 10:09 AM:
-
Some more thoughts. Each key value pair in the log is
{code}
// A record in the log
{ long txid : BlockDeletionTransaction tx}
// where the value is (protobuf message)
BlockDeletionTransaction {
long txid;
String containerName;
List blockIDs;
}
{code}
Following API needs to be implemented
{code}
// Returns a certain size list of transactions
// criteria: each 0<= processedCount < 5
// We may need to maintain a position while scanning the log
// to avoid repetitively scanning a certain range of records
List getBlockDeletionTransactions(int count);
// Increment processedCount for given list of transactions by 1
// if count > 5, reset to -1
void incrementProcessedCount(List txIDs);
// Delete transactions from the log based on their IDs
void commitTransactions(List txIDs);
{code}
Please let me know if this makes sense.
Thanks
was (Author: cheersyang):
Some more thoughts. Each key value pair in the log is
{code}
{ long txid : BlockDeletionTransaction tx}
// where the value is (protobuf message)
BlockDeletionTransaction {
long txid;
String containerName;
List blockIDs;
}
{code}
Following API needs to be implemented
{code}
// Returns a certain size list of transactions
// criteria: each 0<= processedCount < 5
// We may need to maintain a position while scanning the log
// to avoid repetitively scanning a certain range of records
List getBlockDeletionTransactions(int count);
// Increment processedCount for given list of transactions by 1
// if count > 5, reset to -1
void incrementProcessedCount(List txIDs);
// Delete transactions from the log based on their IDs
void commitTransactions(List txIDs);
{code}
Please let me know if this makes sense.
Thanks
> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -
>
> Key: HDFS-12283
> URL: https://issues.apache.org/jira/browse/HDFS-12283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container
> blocks which are under deletion. It maintains info about under-deletion
> container blocks that notified by KSM, and the state how it is processed. We
> can use RocksDB to implement the 1st version of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and
> multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode,
> it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
> means the transaction eventually failed after some retries, 5 is the max
> number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with
> RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
[
https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119663#comment-16119663
]
Weiwei Yang commented on HDFS-12283:
Some more thoughts. Each key value pair in the log is
{code}
{ long txid : BlockDeletionTransaction tx}
// where the value is (protobuf message)
BlockDeletionTransaction {
long txid;
String containerName;
List blockIDs;
}
{code}
Following API needs to be implemented
{code}
// Returns a certain size list of transactions
// criteria: each 0<= processedCount < 5
// We may need to maintain a position while scanning the log
// to avoid repetitively scanning a certain range of records
List getBlockDeletionTransactions(int count);
// Increment processedCount for given list of transactions by 1
// if count > 5, reset to -1
void incrementProcessedCount(List txIDs);
// Delete transactions from the log based on their IDs
void commitTransactions(List txIDs);
{code}
Please let me know if this makes sense.
Thanks
> Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
> -
>
> Key: HDFS-12283
> URL: https://issues.apache.org/jira/browse/HDFS-12283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> The DeletedBlockLog is a persisted log in SCM to keep tracking container
> blocks which are under deletion. It maintains info about under-deletion
> container blocks that notified by KSM, and the state how it is processed. We
> can use RocksDB to implement the 1st version of the log, the schema looks like
> ||TxID||ContainerName||Block List||ProcessedCount||
> |0|c1|b1,b2,b3|0|
> |1|c2|b1|3|
> |2|c2|b2, b3|-1|
> Some explanations
> # TxID is an incremental long value transaction ID for ONE container and
> multiple blocks
> # Container name is the name of the container
> # Block list is a list of block IDs
> # ProcessedCount is the number of times SCM has sent this record to datanode,
> it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
> means the transaction eventually failed after some retries, 5 is the max
> number times of retries.
> We need to define {{DeletedBlockLog}} as an interface and implement this with
> RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
Weiwei Yang created HDFS-12283:
--
Summary: Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
Key: HDFS-12283
URL: https://issues.apache.org/jira/browse/HDFS-12283
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone, scm
Reporter: Weiwei Yang
Assignee: Weiwei Yang
The DeletedBlockLog is a persisted log in SCM to keep tracking container blocks
which are under deletion. It maintains info about under-deletion container
blocks that notified by KSM, and the state how it is processed. We can use
RocksDB to implement the 1st version of the log, the schema looks like
||TxID||ContainerName||Block List||ProcessedCount||
|0|c1|b1,b2,b3|0|
|1|c2|b1|3|
|2|c2|b2, b3|-1|
Some explanations
# TxID is an incremental long value transaction ID for ONE container and
multiple blocks
# Container name is the name of the container
# Block list is a list of block IDs
# ProcessedCount is the number of times SCM has sent this record to datanode,
it represents the "state" of the transaction, it is in range of \[-1, 5\], -1
means the transaction eventually failed after some retries, 5 is the max number
times of retries.
We need to define {{DeletedBlockLog}} as an interface and implement this with
RocksDB {{MetadataStore}} as the first version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12268) Ozone: Add metrics for pending storage container requests
[
https://issues.apache.org/jira/browse/HDFS-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-12268:
-
Attachment: HDFS-12268-HDFS-7240.002.patch
Thanks [~anu] for the review!
bq. Do you think we can maintain average time per op. So we can print out the
performance from the client point of view?
I like this idea, :). The new patch will addresses this.
bq. There is also an added detail that from the ozone-client perspective we
create one XcieverClient per pipeline( a set of datanodes). So when we close
the XcieverClient, we should probably collect this info.
I;m a little confused on this comment. What's the meaning of {{this info}} here?
Attach the updated patch.
> Ozone: Add metrics for pending storage container requests
> -
>
> Key: HDFS-12268
> URL: https://issues.apache.org/jira/browse/HDFS-12268
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-12268-HDFS-7240.001.patch,
> HDFS-12268-HDFS-7240.002.patch
>
>
> As storage container async interface has been supported after HDFS-11580, we
> need to keep an eye on the queue depth of pending container requests. It can
> help us better found if there are some performance problems.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119602#comment-16119602 ] Brahma Reddy Battula edited comment on HDFS-12157 at 8/9/17 9:09 AM: - will hold for commit till [~kihwal] review once. Hi [~kihwal] can you please check once..? thanks was (Author: brahmareddy): will hold for commit till [~kihwal] review once. > Do fsyncDirectory(..) outside of FSDataset lock > --- > > Key: HDFS-12157 > URL: https://issues.apache.org/jira/browse/HDFS-12157 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HDFS-12157-01.patch, HDFS-12157-branch-2-01.patch, > HDFS-12157-branch-2.7-01.patch, HDFS-12157-branch-2.7-01.patch > > > HDFS-5042 introduced fsyncDirectory(..) to save blocks from power failure. > Do it outside of FSDataset lock to avoid overall performance degradation if > disk takes more time to sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119602#comment-16119602 ] Brahma Reddy Battula edited comment on HDFS-12157 at 8/9/17 9:03 AM: - will hold for commit till [~kihwal] review once. was (Author: brahmareddy): will hold for commit till [~kihwal] review once..? > Do fsyncDirectory(..) outside of FSDataset lock > --- > > Key: HDFS-12157 > URL: https://issues.apache.org/jira/browse/HDFS-12157 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HDFS-12157-01.patch, HDFS-12157-branch-2-01.patch, > HDFS-12157-branch-2.7-01.patch, HDFS-12157-branch-2.7-01.patch > > > HDFS-5042 introduced fsyncDirectory(..) to save blocks from power failure. > Do it outside of FSDataset lock to avoid overall performance degradation if > disk takes more time to sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
