[jira] [Commented] (HDFS-12166) Do not deprecate HTTPFS_TEMP
[
https://issues.apache.org/jira/browse/HDFS-12166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094241#comment-16094241
]
Hadoop QA commented on HDFS-12166:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
4s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
42s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12166 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878098/HDFS-12166.002.patch |
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux d685c6618db5 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c21c260 |
| Default Java | 1.8.0_131 |
| shellcheck | v0.4.6 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20345/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U:
hadoop-hdfs-project/hadoop-hdfs-httpfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20345/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Do not deprecate HTTPFS_TEMP
>
>
> Key: HDFS-12166
> URL: https://issues.apache.org/jira/browse/HDFS-12166
> Project:
[jira] [Updated] (HDFS-12166) Do not deprecate HTTPFS_TEMP
[ https://issues.apache.org/jira/browse/HDFS-12166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-12166: -- Attachment: HDFS-12166.002.patch Patch 002 * Use HTTPFS_HOME if defined > Do not deprecate HTTPFS_TEMP > > > Key: HDFS-12166 > URL: https://issues.apache.org/jira/browse/HDFS-12166 > Project: Hadoop HDFS > Issue Type: Bug > Components: httpfs >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HDFS-12166.001.patch, HDFS-12166.002.patch > > > HDFS-10860 deprecated environment variable HTTPFS_TEMP by mistake. This env > var is still useful when the admin uses a custom temp directory for HttpFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12149) Ozone: RocksDB implementation of ozone metadata store
[
https://issues.apache.org/jira/browse/HDFS-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094153#comment-16094153
]
Yuanbo Liu commented on HDFS-12149:
---
[~cheersyang] Thanks for your patch, these are my comments:
LevelDBStore.java
line 165: use close method instead
{code}
close();
{code}
line 128, check null point before closing db
{code}
if(db != null) {
db.close();
}
{code}
RocksDBStore.java
line 237: use close method instead
{code}
close();
{code}
line 267: I don't have much experience in RocksDB, what if iterator doesn't
have next or prev?
> Ozone: RocksDB implementation of ozone metadata store
> -
>
> Key: HDFS-12149
> URL: https://issues.apache.org/jira/browse/HDFS-12149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12149-HDFS-7240.001.patch,
> HDFS-12149-HDFS-7240.002.patch
>
>
> HDFS-12069 added a general interface for ozone metadata store, we already
> have a leveldb implementation, this JIRA is to track the work of rocksdb
> implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11936) Ozone: TestNodeManager times out before it is able to find all nodes
[
https://issues.apache.org/jira/browse/HDFS-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094151#comment-16094151
]
Weiwei Yang commented on HDFS-11936:
I see. Thanks. Can you think about a robust way to check if all heartbeats are
processed?
> Ozone: TestNodeManager times out before it is able to find all nodes
>
>
> Key: HDFS-11936
> URL: https://issues.apache.org/jira/browse/HDFS-11936
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yuanbo Liu
> Attachments: HDFS-11936-HDFS-7240.001.patch
>
>
> During the pre-commit build of
> https://builds.apache.org/job/PreCommit-HDFS-Build/19795/testReport/
> we detected that a test in TestNodeManager is failing. Probably due to the
> fact that we need more time to execute this test in jenkins. This might be
> related to HDFS-11919
> The test failure report follows.
> ==
> {noformat}
> Regression
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport
> Failing for the past 1 build (Since Failed#19795 )
> Took 0.51 sec.
> Error Message
> expected:<2> but was:<18000>
> Stacktrace
> java.lang.AssertionError: expected:<2> but was:<18000>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport(TestNodeManager.java:972)
> Standard Output
> 2017-06-06 13:45:30,909 [main] INFO - Data node with ID:
> 732ebd32-a926-44c5-afbb-c9f87513a67c Registered.
> 2017-06-06 13:45:30,937 [main] INFO - Data node with ID:
> 6860fd5d-94dc-4ba8-acd0-41cc3fa7232d Registered.
> 2017-06-06 13:45:30,971 [main] INFO - Data node with ID:
> cad7174c-204c-4806-b3af-c874706d4bd9 Registered.
> 2017-06-06 13:45:30,996 [main] INFO - Data node with ID:
> 0130a672-719d-4b68-9a1e-13046f4281ff Registered.
> 2017-06-06 13:45:31,021 [main] INFO - Data node with ID:
> 8d9ea5d4-6752-48d4-9bf0-adb0bd1a651a Registered.
> 2017-06-06 13:45:31,046 [main] INFO - Data node with ID:
> f122e372-5a38-476b-97dc-5ae449190485 Registered.
> 2017-06-06 13:45:31,071 [main] INFO - Data node with ID:
> 5750eb03-c1ac-4b3a-bc59-c4d9481e245b Registered.
> 2017-06-06 13:45:31,097 [main] INFO - Data node with ID:
> aa2d90a1-9e85-41f8-a4e5-35c7d2ed7299 Registered.
> 2017-06-06 13:45:31,122 [main] INFO - Data node with ID:
> 5e52bf5c-7050-4fc9-bf10-0e52650229ee Registered.
> 2017-06-06 13:45:31,147 [main] INFO - Data node with ID:
> eaac7b8f-a556-4afc-9163-7309f7ccea18 Registered.
> 2017-06-06 13:45:31,224 [SCM Heartbeat Processing Thread - 0] INFO -
> Current Thread is interrupted, shutting down HB processing thread for Node
> Manager.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12149) Ozone: RocksDB implementation of ozone metadata store
[
https://issues.apache.org/jira/browse/HDFS-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094142#comment-16094142
]
Hadoop QA commented on HDFS-12149:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.ozone.container.replication.TestContainerReplicationManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| Timed out junit tests |
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12149 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878082/HDFS-12149-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 82e645af1207 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b3a7f3b |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20344/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20344/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20344/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This me
[jira] [Commented] (HDFS-11936) Ozone: TestNodeManager times out before it is able to find all nodes
[
https://issues.apache.org/jira/browse/HDFS-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094113#comment-16094113
]
Yuanbo Liu commented on HDFS-11936:
---
[~cheersyang] I'm afraid not. think about this order
{quote}
DN1: send heartBeat
DN2: send heartBeat
HeartbBeat queue size is 2
Thread to handle DN1 report
Thread to handle DN2 report
HeartBeat queue size is empty.
DN3: send heartBeat
.
{quote}
As you can see, heartbeat queue has the chance to be empty, but at this time,
not all the DN's reports are processed, and the test case will fail again.
> Ozone: TestNodeManager times out before it is able to find all nodes
>
>
> Key: HDFS-11936
> URL: https://issues.apache.org/jira/browse/HDFS-11936
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yuanbo Liu
> Attachments: HDFS-11936-HDFS-7240.001.patch
>
>
> During the pre-commit build of
> https://builds.apache.org/job/PreCommit-HDFS-Build/19795/testReport/
> we detected that a test in TestNodeManager is failing. Probably due to the
> fact that we need more time to execute this test in jenkins. This might be
> related to HDFS-11919
> The test failure report follows.
> ==
> {noformat}
> Regression
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport
> Failing for the past 1 build (Since Failed#19795 )
> Took 0.51 sec.
> Error Message
> expected:<2> but was:<18000>
> Stacktrace
> java.lang.AssertionError: expected:<2> but was:<18000>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport(TestNodeManager.java:972)
> Standard Output
> 2017-06-06 13:45:30,909 [main] INFO - Data node with ID:
> 732ebd32-a926-44c5-afbb-c9f87513a67c Registered.
> 2017-06-06 13:45:30,937 [main] INFO - Data node with ID:
> 6860fd5d-94dc-4ba8-acd0-41cc3fa7232d Registered.
> 2017-06-06 13:45:30,971 [main] INFO - Data node with ID:
> cad7174c-204c-4806-b3af-c874706d4bd9 Registered.
> 2017-06-06 13:45:30,996 [main] INFO - Data node with ID:
> 0130a672-719d-4b68-9a1e-13046f4281ff Registered.
> 2017-06-06 13:45:31,021 [main] INFO - Data node with ID:
> 8d9ea5d4-6752-48d4-9bf0-adb0bd1a651a Registered.
> 2017-06-06 13:45:31,046 [main] INFO - Data node with ID:
> f122e372-5a38-476b-97dc-5ae449190485 Registered.
> 2017-06-06 13:45:31,071 [main] INFO - Data node with ID:
> 5750eb03-c1ac-4b3a-bc59-c4d9481e245b Registered.
> 2017-06-06 13:45:31,097 [main] INFO - Data node with ID:
> aa2d90a1-9e85-41f8-a4e5-35c7d2ed7299 Registered.
> 2017-06-06 13:45:31,122 [main] INFO - Data node with ID:
> 5e52bf5c-7050-4fc9-bf10-0e52650229ee Registered.
> 2017-06-06 13:45:31,147 [main] INFO - Data node with ID:
> eaac7b8f-a556-4afc-9163-7309f7ccea18 Registered.
> 2017-06-06 13:45:31,224 [SCM Heartbeat Processing Thread - 0] INFO -
> Current Thread is interrupted, shutting down HB processing thread for Node
> Manager.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12149) Ozone: RocksDB implementation of ozone metadata store
[ https://issues.apache.org/jira/browse/HDFS-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12149: --- Attachment: HDFS-12149-HDFS-7240.002.patch > Ozone: RocksDB implementation of ozone metadata store > - > > Key: HDFS-12149 > URL: https://issues.apache.org/jira/browse/HDFS-12149 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12149-HDFS-7240.001.patch, > HDFS-12149-HDFS-7240.002.patch > > > HDFS-12069 added a general interface for ozone metadata store, we already > have a leveldb implementation, this JIRA is to track the work of rocksdb > implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12149) Ozone: RocksDB implementation of ozone metadata store
[ https://issues.apache.org/jira/browse/HDFS-12149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094077#comment-16094077 ] Weiwei Yang commented on HDFS-12149: Hi [~anu], [~aw] RocksDB 5.5.5 is released, with license updated to apache 2, are we free to go with this patch (of course I am going to update the pom to 5.5.5 first :) ) ? > Ozone: RocksDB implementation of ozone metadata store > - > > Key: HDFS-12149 > URL: https://issues.apache.org/jira/browse/HDFS-12149 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12149-HDFS-7240.001.patch > > > HDFS-12069 added a general interface for ozone metadata store, we already > have a leveldb implementation, this JIRA is to track the work of rocksdb > implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12115) Ozone: SCM: Add queryNode RPC Call
[ https://issues.apache.org/jira/browse/HDFS-12115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094070#comment-16094070 ] Weiwei Yang commented on HDFS-12115: Thanks [~anu], +1 for the patch once the checkstyle issues are fixed. And one more question, there seems to have several UT failures, are they related to this patch? > Ozone: SCM: Add queryNode RPC Call > -- > > Key: HDFS-12115 > URL: https://issues.apache.org/jira/browse/HDFS-12115 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12115-HDFS-7240.001.patch, > HDFS-12115-HDFS-7240.002.patch, HDFS-12115-HDFS-7240.003.patch, > HDFS-12115-HDFS-7240.004.patch, HDFS-12115-HDFS-7240.005.patch, > HDFS-12115-HDFS-7240.006.patch > > > Add queryNode RPC to Storage container location protocol. This allows > applications like SCM CLI to get the list of nodes in various states, like > Healthy, live or Dead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094067#comment-16094067 ] Weiwei Yang commented on HDFS-12163: Very interesting, I observed a number of leaks as well in the past. Most recent one I am working on is HDFS-12098, that problem was caused by thread leak too. [~szetszwo] are you working on a fix? If not, I can look into this problem this weekend. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12166) Do not deprecate HTTPFS_TEMP
[
https://issues.apache.org/jira/browse/HDFS-12166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094059#comment-16094059
]
Hadoop QA commented on HDFS-12166:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
4s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
25s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12166 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878075/HDFS-12166.001.patch |
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall findbugs checkstyle |
| uname | Linux 257622b0f532 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c21c260 |
| Default Java | 1.8.0_131 |
| shellcheck | v0.4.6 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20343/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U:
hadoop-hdfs-project/hadoop-hdfs-httpfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20343/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Do not deprecate HTTPFS_TEMP
>
>
> Key: HDFS-12166
> URL: https://issues.apache.org/jira/browse/HDFS-12166
> Project:
[jira] [Work started] (HDFS-9507) LeaseRenewer Logging Under-Reporting
[
https://issues.apache.org/jira/browse/HDFS-9507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-9507 started by Bharat Viswanadham.
> LeaseRenewer Logging Under-Reporting
>
>
> Key: HDFS-9507
> URL: https://issues.apache.org/jira/browse/HDFS-9507
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: BELUGA BEHR
>Assignee: Bharat Viswanadham
>Priority: Minor
>
> Why is it that in LeaseRenewer#run() failures to renew a lease on a file are
> reported with "warn" level logging, but in LeaseRenewer#renew() it is
> reported with a "debug" level warn?
> In LeaseRenewer#renew(), if the method renewLease() returns 'false' then the
> problem is silently discarded (continue, no Exception is thrown) and the next
> client in the list tries to renew.
> {code:title=LeaseRenewer.java|borderStyle=solid}
> private void run(final int id) throws InterruptedException {
> ...
> try {
> renew();
> lastRenewed = Time.monotonicNow();
> } catch (SocketTimeoutException ie) {
> LOG.warn("Failed to renew lease for " + clientsString() + " for "
> + (elapsed/1000) + " seconds. Aborting ...", ie);
> synchronized (this) {
> while (!dfsclients.isEmpty()) {
> DFSClient dfsClient = dfsclients.get(0);
> dfsClient.closeAllFilesBeingWritten(true);
> closeClient(dfsClient);
> }
> //Expire the current LeaseRenewer thread.
> emptyTime = 0;
> }
> break;
> } catch (IOException ie) {
> LOG.warn("Failed to renew lease for " + clientsString() + " for "
> + (elapsed/1000) + " seconds. Will retry shortly ...", ie);
> }
> }
> ...
> }
> private void renew() throws IOException {
> {
>...
> if (!c.renewLease()) {
> LOG.debug("Did not renew lease for client {}", c);
> continue;
> }
>...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-5896) DataXceiver and DFSOutputStream call Thread.setName(), which is not thread safe
[ https://issues.apache.org/jira/browse/HDFS-5896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-5896 started by Bharat Viswanadham. > DataXceiver and DFSOutputStream call Thread.setName(), which is not thread > safe > --- > > Key: HDFS-5896 > URL: https://issues.apache.org/jira/browse/HDFS-5896 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hiroshi Ikeda >Assignee: Bharat Viswanadham >Priority: Trivial > > org.apache.hadoop.hdfs.server.datanode.DataXceiver and > org.apache.hadoop.hdfs.DFSOutputStream renames running threads, but > Thread.setName() is not thread safe. Thread.setName() is currently intended > to call before running the thread. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11472) Fix inconsistent replica size after a data pipeline failure
[
https://issues.apache.org/jira/browse/HDFS-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094050#comment-16094050
]
Hadoop QA commented on HDFS-11472:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 109 unchanged - 1 fixed = 110 total (was 110) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11472 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878064/HDFS-11472.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 406f8bb6f1f5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c21c260 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20342/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20342/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20342/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20342/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-
[jira] [Comment Edited] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093590#comment-16093590
]
Konstantin Shvachko edited comment on HDFS-11896 at 7/20/17 1:13 AM:
-
Hey [~brahmareddy],
* in your trunk patch you correctly reset nonDfsUsed to 0, but not other
fields. I was thinking we could factor out the part of
{{updateHeartbeatState()}} that sets the "total" fields excluding the update
times and use it in {{resetBlocks()}}. That way we know all fields are updated
even if somebody adds new ones.
* In the 2.7 patch you follow the HDFS-9034 path introducing
{{updateDnStat()}}, which will cause test failures as HDFS-9034 did. I
recommend instead to swap lines in {{HeartbeatManager.register()}} without
modifying {{addDatanode()}}.
Like this:
{code}
synchronized void register(final DatanodeDescriptor d) {
if (!d.isAlive) {
- addDatanode(d);
d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null);
+ addDatanode(d);
}
}
{code}
was (Author: shv):
Hey [~brahmareddy],
* in your trunk patch you correctly reset nonDfsUsed to 0, but not other
fields. I was thinking we could factor out the part of
{{updateHeartbeatState()}} that sets the "total" fields excluding the update
times and use it in {{resetBlocks()}}. That way we know all fields are updated
even if somebody adds new ones.
* In the 2.7 patch you follow the HDFS-9034 path introducing
{{updateDnStat()}}, which will cause test failures as HDFS-9034 did. I
recommend instead to swap lines in {{HeartbeatManager.register()}} without
modifying {{addDatanode()}}.
Like this:
{code}
synchronized void register(final DatanodeDescriptor d) {
if (!d.isAlive) {
- addDatanode(d);
- d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null);
+ d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null);
+ addDatanode(d);
}
}
{code}
> Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
> ---
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Labels: release-blocker
> Attachments: HDFS-11896-002.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093590#comment-16093590
]
Konstantin Shvachko edited comment on HDFS-11896 at 7/20/17 1:12 AM:
-
Hey [~brahmareddy],
* in your trunk patch you correctly reset nonDfsUsed to 0, but not other
fields. I was thinking we could factor out the part of
{{updateHeartbeatState()}} that sets the "total" fields excluding the update
times and use it in {{resetBlocks()}}. That way we know all fields are updated
even if somebody adds new ones.
* In the 2.7 patch you follow the HDFS-9034 path introducing
{{updateDnStat()}}, which will cause test failures as HDFS-9034 did. I
recommend instead to swap lines in {{HeartbeatManager.register()}} without
modifying {{addDatanode()}}.
Like this:
{code}
synchronized void register(final DatanodeDescriptor d) {
if (!d.isAlive) {
- addDatanode(d);
- d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null);
+ d.updateHeartbeatState(StorageReport.EMPTY_ARRAY, 0L, 0L, 0, 0, null);
+ addDatanode(d);
}
}
{code}
was (Author: shv):
Hey [~brahmareddy],
* in your trunk patch you correctly reset nonDfsUsed to 0, but not other
fields. I was thinking we could factor out the part of
{{updateHeartbeatState()}} that sets the "total" fields excluding the update
times and use it in {{resetBlocks()}}. That way we know all fields are updated
even if somebody adds new ones.
* In the 2.7 patch you follow the HDFS-9034 path introducing
{{updateDnStat()}}, which will cause test failures as HDFS-9034 did. I
recommend instead to swap lines in {{HeartbeatManager.register()}} without
modifying {{addDatanode()}}.
> Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
> ---
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Labels: release-blocker
> Attachments: HDFS-11896-002.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11472) Fix inconsistent replica size after a data pipeline failure
[
https://issues.apache.org/jira/browse/HDFS-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094040#comment-16094040
]
Konstantin Shvachko commented on HDFS-11472:
+1 on the latest patch.
Patches for other branches needed as well. It is not applying cleanly even to
branch-2.
> Fix inconsistent replica size after a data pipeline failure
> ---
>
> Key: HDFS-11472
> URL: https://issues.apache.org/jira/browse/HDFS-11472
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-11472.001.patch, HDFS-11472.002.patch,
> HDFS-11472.003.patch, HDFS-11472.004.patch, HDFS-11472.005.patch,
> HDFS-11472.testcase.patch
>
>
> We observed a case where a replica's on disk length is less than acknowledged
> length, breaking the assumption in recovery code.
> {noformat}
> 2017-01-08 01:41:03,532 WARN
> org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed to
> obtain replica info for block
> (=BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394519586) from
> datanode (=DatanodeInfoWithStorage[10.204.138.17:1004,null,null])
> java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: getBytesOnDisk() <
> getVisibleLength(), rip=ReplicaBeingWritten, blk_2526438952_1101394519586, RBW
> getNumBytes() = 27530
> getBytesOnDisk() = 27006
> getVisibleLength()= 27268
> getVolume() = /data/6/hdfs/datanode/current
> getBlockFile()=
> /data/6/hdfs/datanode/current/BP-947993742-10.204.0.136-1362248978912/current/rbw/blk_2526438952
> bytesAcked=27268
> bytesOnDisk=27006
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2284)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2260)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2566)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.callInitReplicaRecovery(DataNode.java:2577)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.recoverBlock(DataNode.java:2645)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.access$400(DataNode.java:245)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$5.run(DataNode.java:2551)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> It turns out that if an exception is thrown within
> {{BlockReceiver#receivePacket}}, the in-memory replica on disk length may not
> be updated, but the data is written to disk anyway.
> For example, here's one exception we observed
> {noformat}
> 2017-01-08 01:40:59,512 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for
> BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394499067
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.adjustCrcChannelPosition(FsDatasetImpl.java:1484)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.adjustCrcFilePosition(BlockReceiver.java:994)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:670)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:797)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There are potentially other places and causes where an exception is thrown
> within {{BlockReceiver#receivePacket}}, so it may not make much sense to
> alleviate it for this particular exception. Instead, we should improve
> replica recovery code to handle the case where ondisk size is less than
> acknowledged size, and update in-memory checksum accordingly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12115) Ozone: SCM: Add queryNode RPC Call
[
https://issues.apache.org/jira/browse/HDFS-12115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094028#comment-16094028
]
Hadoop QA commented on HDFS-12115:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 12 new
+ 153 unchanged - 0 fixed = 165 total (was 153) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.ozone.container.replication.TestContainerReplicationManager |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.ozone.container.placement.TestContainerPlacement |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12115 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878057/HDFS-12115-HDFS-7240.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml
[jira] [Updated] (HDFS-12166) Do not deprecate HTTPFS_TEMP
[ https://issues.apache.org/jira/browse/HDFS-12166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-12166: -- Attachment: HDFS-12166.001.patch Patch 001 * Un-deprecate HTTPFS_TEMP * Set system property httpfs.temp.dir > Do not deprecate HTTPFS_TEMP > > > Key: HDFS-12166 > URL: https://issues.apache.org/jira/browse/HDFS-12166 > Project: Hadoop HDFS > Issue Type: Bug > Components: httpfs >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HDFS-12166.001.patch > > > HDFS-10860 deprecated environment variable HTTPFS_TEMP by mistake. This env > var is still useful when the admin uses a custom temp directory for HttpFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12166) Do not deprecate HTTPFS_TEMP
[ https://issues.apache.org/jira/browse/HDFS-12166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-12166: -- Status: Patch Available (was: Open) > Do not deprecate HTTPFS_TEMP > > > Key: HDFS-12166 > URL: https://issues.apache.org/jira/browse/HDFS-12166 > Project: Hadoop HDFS > Issue Type: Bug > Components: httpfs >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HDFS-12166.001.patch > > > HDFS-10860 deprecated environment variable HTTPFS_TEMP by mistake. This env > var is still useful when the admin uses a custom temp directory for HttpFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11799) Introduce a config to allow setting up write pipeline with fewer nodes than replication factor
[
https://issues.apache.org/jira/browse/HDFS-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094015#comment-16094015
]
Yongjun Zhang commented on HDFS-11799:
--
HI [~brahmareddy],
Thanks for the updated patch and really worry for the delayed review. It in
general looks pretty good. I have some comments about cosmetics things:
1. Suggest to change
{code}
DFSClient.LOG.warn(
"Failed to add a new datanode for write pipeline, minimum block "
+ "replication:"
+ dfsClient.dtpReplaceDatanodeOnFailureReplication
+ ", good datanode size: " + nodes.length, ioe);
{code}
to
{code}
DFSClient.LOG.warn(
"Failed to find a new datanode to add to the write pipeline, "
" continue to write to the pipeline with " + nodes.length
" nodes since it's bigger than minimum replication: "
+ dfsClient.dtpReplaceDatanodeOnFailureReplication
+ " configued by "
+
HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION,
+ ".", ioe);
{code}
2. Seems we don't need to add two new entries and call them deprecated:
{code}
@Deprecated
public static final String
DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_REPLICATION =
HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION;
@Deprecated
public static final short
DFS_CLIENT_WRITE_REPLACE_DATANODE_ON_FAILURE_REPLICATION_DEFAULT =
HdfsClientConfigKeys.BlockWrite.ReplaceDatanodeOnFailure.REPLICATION_DEFAULT;
{code}
So we can drop them right?
3. in hdfs-default.xml, what about change it to:
{code}
The minimum number of replications that are needed to not to fail
the write pipeline if new datanodes can not be found to replace
failed datanodes (could be due to network failure) in the write pipeline.
If the number of the remaining datanodes in the write pipeline is greater
than or
equal to this property value, continue writing to the remaining nodes.
Otherwise throw exception.
If this is set to 0 or a negative number, an exception will be thrown
when a replacement can not be found.
See also dfs.client.block.write.replace-datanode-on-failure.policy
{code}
4. Thanks for the comprehensive tests. I notice that the comment of test code
sometimes don't match the code, e.g.
{code}
54 /** Test fail all the datanodes except first in the pipeline. */
255 @Test public void testWithOnlyLastDatanodeIsAlive() throws Exception {
{code}
and
{code}
317 /** Test fail all the datanodes except first in the pipeline. */
318 @Test
319 public void
testLessNumberOfLiveDatanodesThanWriteReplaceDatanodeOnFailureRF()
{code}
5. Possible to consolidate simliar part of the tests into one helper method
with a parameter to indicate which
DNs to stop, to avoid redundant code?
6. Seems even for the test that exception is thrown due to failure to find new
datanode, the
result file can still be verified to have the expected full content with
verifyFileContent()?
> Introduce a config to allow setting up write pipeline with fewer nodes than
> replication factor
> --
>
> Key: HDFS-11799
> URL: https://issues.apache.org/jira/browse/HDFS-11799
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-11799-002.patch, HDFS-11799-003.patch,
> HDFS-11799.patch
>
>
> During pipeline recovery, if not enough DNs can be found, if
> dfs.client.block.write.replace-datanode-on-failure.best-effort
> is enabled, we let the pipeline to continue, even if there is a single DN.
> Similarly, when we create the write pipeline initially, if for some reason we
> can't find enough DNs, we can have a similar config to enable writing with a
> single DN.
> More study will be done.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12166) Do not deprecate HTTPFS_TEMP
[ https://issues.apache.org/jira/browse/HDFS-12166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-12166: -- Description: HDFS-10860 deprecated environment variable HTTPFS_TEMP by mistake. This env var is still useful when the admin uses a custom temp directory for HttpFS. (was: HDFS-10860 deprecated environment variable HTTPFS_TEMP by mistake.) > Do not deprecate HTTPFS_TEMP > > > Key: HDFS-12166 > URL: https://issues.apache.org/jira/browse/HDFS-12166 > Project: Hadoop HDFS > Issue Type: Bug > Components: httpfs >Affects Versions: 3.0.0-alpha4 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > > HDFS-10860 deprecated environment variable HTTPFS_TEMP by mistake. This env > var is still useful when the admin uses a custom temp directory for HttpFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12166) Do not deprecate HTTPFS_TEMP
John Zhuge created HDFS-12166: - Summary: Do not deprecate HTTPFS_TEMP Key: HDFS-12166 URL: https://issues.apache.org/jira/browse/HDFS-12166 Project: Hadoop HDFS Issue Type: Bug Components: httpfs Affects Versions: 3.0.0-alpha4 Reporter: John Zhuge Assignee: John Zhuge Priority: Minor HDFS-10860 deprecated environment variable HTTPFS_TEMP by mistake. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12165) getSnapshotDiffReport throws NegativeArraySizeException for very large snapshot diff summary
Wei-Chiu Chuang created HDFS-12165:
--
Summary: getSnapshotDiffReport throws NegativeArraySizeException
for very large snapshot diff summary
Key: HDFS-12165
URL: https://issues.apache.org/jira/browse/HDFS-12165
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Wei-Chiu Chuang
For a really large snapshot diff, getSnapshotDiffReport throws
NegativeArraySizeException
{noformat}
2017-07-19 11:14:16,415 WARN org.apache.hadoop.ipc.Server: Error serializing
call response for call
org.apache.hadoop.hdfs.protocol.ClientProtocol.getSnapshotDiffReport
from 10.17.211.10:58223 Call#0 Retry#0
java.lang.NegativeArraySizeException
at
com.google.protobuf.CodedOutputStream.newInstance(CodedOutputStream.java:105)
at
com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:87)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseWrapper.write(ProtobufRpcEngine.java:468)
at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2410)
at org.apache.hadoop.ipc.Server.access$500(Server.java:134)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2182)
{noformat}
This particular snapshot diff contains more than 25 million different file
system objects, and which means the serialized response can be more than 2GB,
overflowing protobuf length calculation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11472) Fix inconsistent replica size after a data pipeline failure
[
https://issues.apache.org/jira/browse/HDFS-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093959#comment-16093959
]
Erik Krogen commented on HDFS-11472:
Thanks for the review [~shv]. Agreed with you on the
{{s/numBytes/bytesOnDisk/}} change. I added a Javadoc comment to the test as
well. Attached v005 patch.
> Fix inconsistent replica size after a data pipeline failure
> ---
>
> Key: HDFS-11472
> URL: https://issues.apache.org/jira/browse/HDFS-11472
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-11472.001.patch, HDFS-11472.002.patch,
> HDFS-11472.003.patch, HDFS-11472.004.patch, HDFS-11472.005.patch,
> HDFS-11472.testcase.patch
>
>
> We observed a case where a replica's on disk length is less than acknowledged
> length, breaking the assumption in recovery code.
> {noformat}
> 2017-01-08 01:41:03,532 WARN
> org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed to
> obtain replica info for block
> (=BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394519586) from
> datanode (=DatanodeInfoWithStorage[10.204.138.17:1004,null,null])
> java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: getBytesOnDisk() <
> getVisibleLength(), rip=ReplicaBeingWritten, blk_2526438952_1101394519586, RBW
> getNumBytes() = 27530
> getBytesOnDisk() = 27006
> getVisibleLength()= 27268
> getVolume() = /data/6/hdfs/datanode/current
> getBlockFile()=
> /data/6/hdfs/datanode/current/BP-947993742-10.204.0.136-1362248978912/current/rbw/blk_2526438952
> bytesAcked=27268
> bytesOnDisk=27006
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2284)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2260)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2566)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.callInitReplicaRecovery(DataNode.java:2577)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.recoverBlock(DataNode.java:2645)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.access$400(DataNode.java:245)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$5.run(DataNode.java:2551)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> It turns out that if an exception is thrown within
> {{BlockReceiver#receivePacket}}, the in-memory replica on disk length may not
> be updated, but the data is written to disk anyway.
> For example, here's one exception we observed
> {noformat}
> 2017-01-08 01:40:59,512 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for
> BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394499067
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.adjustCrcChannelPosition(FsDatasetImpl.java:1484)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.adjustCrcFilePosition(BlockReceiver.java:994)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:670)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:797)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There are potentially other places and causes where an exception is thrown
> within {{BlockReceiver#receivePacket}}, so it may not make much sense to
> alleviate it for this particular exception. Instead, we should improve
> replica recovery code to handle the case where ondisk size is less than
> acknowledged size, and update in-memory checksum accordingly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11472) Fix inconsistent replica size after a data pipeline failure
[
https://issues.apache.org/jira/browse/HDFS-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-11472:
---
Attachment: HDFS-11472.005.patch
> Fix inconsistent replica size after a data pipeline failure
> ---
>
> Key: HDFS-11472
> URL: https://issues.apache.org/jira/browse/HDFS-11472
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-11472.001.patch, HDFS-11472.002.patch,
> HDFS-11472.003.patch, HDFS-11472.004.patch, HDFS-11472.005.patch,
> HDFS-11472.testcase.patch
>
>
> We observed a case where a replica's on disk length is less than acknowledged
> length, breaking the assumption in recovery code.
> {noformat}
> 2017-01-08 01:41:03,532 WARN
> org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed to
> obtain replica info for block
> (=BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394519586) from
> datanode (=DatanodeInfoWithStorage[10.204.138.17:1004,null,null])
> java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: getBytesOnDisk() <
> getVisibleLength(), rip=ReplicaBeingWritten, blk_2526438952_1101394519586, RBW
> getNumBytes() = 27530
> getBytesOnDisk() = 27006
> getVisibleLength()= 27268
> getVolume() = /data/6/hdfs/datanode/current
> getBlockFile()=
> /data/6/hdfs/datanode/current/BP-947993742-10.204.0.136-1362248978912/current/rbw/blk_2526438952
> bytesAcked=27268
> bytesOnDisk=27006
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2284)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2260)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2566)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.callInitReplicaRecovery(DataNode.java:2577)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.recoverBlock(DataNode.java:2645)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.access$400(DataNode.java:245)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$5.run(DataNode.java:2551)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> It turns out that if an exception is thrown within
> {{BlockReceiver#receivePacket}}, the in-memory replica on disk length may not
> be updated, but the data is written to disk anyway.
> For example, here's one exception we observed
> {noformat}
> 2017-01-08 01:40:59,512 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for
> BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394499067
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.adjustCrcChannelPosition(FsDatasetImpl.java:1484)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.adjustCrcFilePosition(BlockReceiver.java:994)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:670)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:797)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There are potentially other places and causes where an exception is thrown
> within {{BlockReceiver#receivePacket}}, so it may not make much sense to
> alleviate it for this particular exception. Instead, we should improve
> replica recovery code to handle the case where ondisk size is less than
> acknowledged size, and update in-memory checksum accordingly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093957#comment-16093957 ] Xiaoyu Yao commented on HDFS-12159: --- Patch looks good to me overall. Just few minor issues: Ozone.proto Line 76: should we add CHAINED replication pipeline as place holder? StorageContainerLocationProtocol.proto Do we have other management APIs to be added later, e.g., list/delete/get Pipleline? StorageContainerLocationProtocolServerSideTranslatorPB.java Line 164: TODO? PipelineManager.java Line 32: NIT: should we call this like createPipeline? Line 38: can you elaborate on the implication of closeCluster()? Does it delete/remove the pipeline? > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11472) Fix inconsistent replica size after a data pipeline failure
[
https://issues.apache.org/jira/browse/HDFS-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093953#comment-16093953
]
Konstantin Shvachko commented on HDFS-11472:
[~xkrogen] this should work correctly. I would change one condition:
{code}
-if (numBytes > bytesAcked) {
+if (bytesOnDisk > bytesAcked) {
{code}
because during physical recovery we do not care about in-memory {{numBytes}},
but rather the actual block file length = {{bytesOnDisk}}.
Also would be good if you could add a JavaDoc comment, explaining what it
tests, as an example of good coding style.
> Fix inconsistent replica size after a data pipeline failure
> ---
>
> Key: HDFS-11472
> URL: https://issues.apache.org/jira/browse/HDFS-11472
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Critical
> Labels: release-blocker
> Attachments: HDFS-11472.001.patch, HDFS-11472.002.patch,
> HDFS-11472.003.patch, HDFS-11472.004.patch, HDFS-11472.testcase.patch
>
>
> We observed a case where a replica's on disk length is less than acknowledged
> length, breaking the assumption in recovery code.
> {noformat}
> 2017-01-08 01:41:03,532 WARN
> org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed to
> obtain replica info for block
> (=BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394519586) from
> datanode (=DatanodeInfoWithStorage[10.204.138.17:1004,null,null])
> java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: getBytesOnDisk() <
> getVisibleLength(), rip=ReplicaBeingWritten, blk_2526438952_1101394519586, RBW
> getNumBytes() = 27530
> getBytesOnDisk() = 27006
> getVisibleLength()= 27268
> getVolume() = /data/6/hdfs/datanode/current
> getBlockFile()=
> /data/6/hdfs/datanode/current/BP-947993742-10.204.0.136-1362248978912/current/rbw/blk_2526438952
> bytesAcked=27268
> bytesOnDisk=27006
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2284)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.initReplicaRecovery(FsDatasetImpl.java:2260)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initReplicaRecovery(DataNode.java:2566)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.callInitReplicaRecovery(DataNode.java:2577)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.recoverBlock(DataNode.java:2645)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.access$400(DataNode.java:245)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode$5.run(DataNode.java:2551)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> It turns out that if an exception is thrown within
> {{BlockReceiver#receivePacket}}, the in-memory replica on disk length may not
> be updated, but the data is written to disk anyway.
> For example, here's one exception we observed
> {noformat}
> 2017-01-08 01:40:59,512 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Exception for
> BP-947993742-10.204.0.136-1362248978912:blk_2526438952_1101394499067
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:269)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.adjustCrcChannelPosition(FsDatasetImpl.java:1484)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.adjustCrcFilePosition(BlockReceiver.java:994)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:670)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:857)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:797)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:244)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There are potentially other places and causes where an exception is thrown
> within {{BlockReceiver#receivePacket}}, so it may not make much sense to
> alleviate it for this particular exception. Instead, we should improve
> replica recovery code to handle the case where ondisk size is less than
> acknowledged size, and update in-memory checksum accordingly.
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HDFS-12115) Ozone: SCM: Add queryNode RPC Call
[ https://issues.apache.org/jira/browse/HDFS-12115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12115: Attachment: HDFS-12115-HDFS-7240.006.patch bq. InProgressPool.java: NIT, line 203, extra space between NodeState and getNodeState Fixed. bq. MockNodeManager.java: line 161-173, it seems this can be replaced by getNodes(nodestate).size() Fixed. bq. Ozone.proto : Add a placeholder for DECOMMISSIONING state? Fixed. bq. SCMNodeManager.java : line 413-435, like you mentioned earlier, a node may have more than 1 state, e.g both HEALTHY The reason why queryNode needs to be able to read multiple states is because it is an RPC and a network call. But getNodeState is a local call and currently only use is to get the liveness of the nodes. If and when we get to automatic membership management, we can make this modification. Since no one is calling it now, I have left it as it is for now. bq. line 491: instead of creating a new list, this can be done in java8 style return currentSet.stream().collect(Collectors.toList()); I presume you are referring to StorageContainerManager.java. The reason why I am not doing it is Line: 479. Since this is an "And", if we find any null sets for any attribute, we return an empty set. With an empty ResultSet, the return is consistent, the same set object which is always non-null. > Ozone: SCM: Add queryNode RPC Call > -- > > Key: HDFS-12115 > URL: https://issues.apache.org/jira/browse/HDFS-12115 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12115-HDFS-7240.001.patch, > HDFS-12115-HDFS-7240.002.patch, HDFS-12115-HDFS-7240.003.patch, > HDFS-12115-HDFS-7240.004.patch, HDFS-12115-HDFS-7240.005.patch, > HDFS-12115-HDFS-7240.006.patch > > > Add queryNode RPC to Storage container location protocol. This allows > applications like SCM CLI to get the list of nodes in various states, like > Healthy, live or Dead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11948) Ozone: change TestRatisManager to check cluster with data
[ https://issues.apache.org/jira/browse/HDFS-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093896#comment-16093896 ] Tsz Wo Nicholas Sze commented on HDFS-11948: No, I suspect it was IntelliJ. Will try running the tests with maven directly. BTW, we should figure out why an 1-datanode uses 400+ threads. It probably is caused by some default conf properties being set to a large value. > Ozone: change TestRatisManager to check cluster with data > - > > Key: HDFS-11948 > URL: https://issues.apache.org/jira/browse/HDFS-11948 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11948-HDFS-7240.20170614.patch > > > TestRatisManager first creates multiple Ratis clusters. Then it changes the > membership and closes some clusters. However, it does not test the clusters > with data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093893#comment-16093893 ] Kuhu Shukla commented on HDFS-5040: --- bq. Also Check for usages of checkSuperuserPrivilege(), wherever this is called, add audit logs for that RPC if not already there.also move checkSuperuserPrivilege() check even before obtaining any lock. Should this be done separately since commands like listCorruptFileBlocks is not an admin command? > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.patch, HDFS-5040.patch, > HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11948) Ozone: change TestRatisManager to check cluster with data
[ https://issues.apache.org/jira/browse/HDFS-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093890#comment-16093890 ] Anu Engineer commented on HDFS-11948: - Thanks for sharing this info. Do you have any sense of where the leak is happening? > Ozone: change TestRatisManager to check cluster with data > - > > Key: HDFS-11948 > URL: https://issues.apache.org/jira/browse/HDFS-11948 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11948-HDFS-7240.20170614.patch > > > TestRatisManager first creates multiple Ratis clusters. Then it changes the > membership and closes some clusters. However, it does not test the clusters > with data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093885#comment-16093885 ] Tsz Wo Nicholas Sze commented on HDFS-12163: All the results above are generated using IntelliJ. I will try running the tests with maven directly. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11948) Ozone: change TestRatisManager to check cluster with data
[ https://issues.apache.org/jira/browse/HDFS-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093867#comment-16093867 ] Tsz Wo Nicholas Sze commented on HDFS-11948: > Also in the profiler, I see around 2025 threads being launched for this > single test. Thought you might be interested in that. [~anu], I just have finished some testing and filed HDFS-12163. It turns out that an 1-datanode MiniOzoneCluster uses 400+ threads for both Ratis and non-Ratis settings. Also, there is a thread leak problem -- some threads used in earlier tests are leaked to later tests. The thread leak explains why there are around 2025 threads showed in the profiler. TestRatisManager uses 5 datanodes. It only uses a similar number of threads as other 5-datanode cases showed in [this comment|https://issues.apache.org/jira/browse/HDFS-12163?focusedCommentId=16093848&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16093848] - testTestRatisManagerNetty | (NETTY,5) | init | 6 | | (NETTY,5) | MiniOzoneCluster | 598 | | (NETTY,5) | ready | 598 | | (NETTY,5) | create RatisCluster | 690 | | (NETTY,5) | close RatisCluster | 722 | | (NETTY,5) | update RatisCluster | 807 | | (NETTY,5) | shutdown | 438 | | (NETTY,5) | sleep | 143 | - testTestRatisManagerGrpc | (GRPC,5) | init | 6 | | (GRPC,5) | MiniOzoneCluster | 600 | | (GRPC,5) | ready | 600 | | (GRPC,5) | create RatisCluster | 621 | | (GRPC,5) | close RatisCluster | 623 | | (GRPC,5) | update RatisCluster | 633 | | (GRPC,5) | shutdown | 257 | | (GRPC,5) | sleep | 137 | > Ozone: change TestRatisManager to check cluster with data > - > > Key: HDFS-11948 > URL: https://issues.apache.org/jira/browse/HDFS-11948 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11948-HDFS-7240.20170614.patch > > > TestRatisManager first creates multiple Ratis clusters. Then it changes the > membership and closes some clusters. However, it does not test the clusters > with data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12155) Ozone : add RocksDB support to DEBUG CLI
[ https://issues.apache.org/jira/browse/HDFS-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12155: -- Attachment: HDFS-12155-HDFS-7240.001.patch Turns out that with very minimal change, the debug cli will be able to read from rockesDB. Post init patch. This one depends on HDFS-12149 to be committed first. > Ozone : add RocksDB support to DEBUG CLI > > > Key: HDFS-12155 > URL: https://issues.apache.org/jira/browse/HDFS-12155 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12155-HDFS-7240.001.patch > > > As we are migrating to replacing LevelDB with RocksDB, we should also add the > support of RocksDB to the debug cli. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12155) Ozone : add RocksDB support to DEBUG CLI
[ https://issues.apache.org/jira/browse/HDFS-12155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12155 started by Chen Liang. - > Ozone : add RocksDB support to DEBUG CLI > > > Key: HDFS-12155 > URL: https://issues.apache.org/jira/browse/HDFS-12155 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-12155-HDFS-7240.001.patch > > > As we are migrating to replacing LevelDB with RocksDB, we should also add the > support of RocksDB to the debug cli. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093852#comment-16093852 ] Tsz Wo Nicholas Sze commented on HDFS-12163: The thread leak is serious in the 5-datanode consecutive run. There are 838 threads remaining at the end. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093848#comment-16093848 ] Tsz Wo Nicholas Sze commented on HDFS-12163: 5-datanode tests - Local handlers with 5 datanode | (local,5) | init | 6 | | (local,5) | MiniOzoneCluster | 589 | | (local,5) | shutdown | 209 | | (local,5) | sleep | 89 | - Distributed handlers with 5 datanode | (distributed,5) | init | 6 | | (distributed,5) | MiniOzoneCluster | 589 | | (distributed,5) | shutdown | 209 | | (distributed,5) | sleep | 89 | - Ratis-Netty handlers with 5 datanode | (NETTY,5) | init | 6 | | (NETTY,5) | MiniOzoneCluster | 600 | | (NETTY,5) | createRatisCluster | 712 | | (NETTY,5) | shutdown | 332 | | (NETTY,5) | sleep | 132 | - Ratis-gRPC handlers with 5 datanode | (GRPC,5) | init | 6 | | (GRPC,5) | MiniOzoneCluster | 596 | | (GRPC,5) | createRatisCluster | 637 | | (GRPC,5) | shutdown | 257 | | (GRPC,5) | sleep | 138 | - All 4 cases above running consecutively. | (distributed,5) | init | 6 | | (distributed,5) | MiniOzoneCluster | 589 | | (distributed,5) | shutdown | 209 | | (distributed,5) | sleep | 90 | | (GRPC,5) | init | 90 | | (GRPC,5) | MiniOzoneCluster | 691 | | (GRPC,5) | createRatisCluster | 740 | | (GRPC,5) | shutdown | 360 | | (GRPC,5) | sleep | 290 | | (local,5) | init | 290 | | (local,5) | MiniOzoneCluster | 907 | | (local,5) | shutdown | 527 | | (local,5) | sleep | 505 | | (NETTY,5) | init | 505 | | (NETTY,5) | MiniOzoneCluster | 1147 | | (NETTY,5) | createRatisCluster | 1269 | | (NETTY,5) | shutdown | 890 | | (NETTY,5) | sleep | 838 | > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093842#comment-16093842 ] Tsz Wo Nicholas Sze commented on HDFS-12163: 1-datanode tests - Ratis-gRPC with 1 datanode | (GRPC,1) | init | 6 | | (GRPC,1) | MiniOzoneCluster | 404 | | (GRPC,1) | createRatisCluster | 413 | | (GRPC,1) | shutdown | 65 | | (GRPC,1) | sleep | 38 | - Ratis-Netty with 1 datanode | (NETTY,1) | init | 6 | | (NETTY,1) | MiniOzoneCluster | 404 | | (NETTY,1) | createRatisCluster | 424 | | (NETTY,1) | shutdown | 76 | | (NETTY,1) | sleep | 33 | - Distributed handlers with 1 datanode | (distributed,1) | init | 6 | | (distributed,1) | MiniOzoneCluster | 402 | | (distributed,1) | shutdown | 54 | | (distributed,1) | sleep | 27 | - Local handlers with 1 datanode | (local,1) | init | 6 | | (local,1) | MiniOzoneCluster | 402 | | (local,1) | shutdown | 54 | | (local,1) | sleep | 27 | All the results above are from separated runs. Below is the result running them together. - All 4 cases above running consecutively. | (distributed,1) | init | 6 | | (distributed,1) | MiniOzoneCluster | 402 | | (distributed,1) | shutdown | 54 | | (distributed,1) | sleep | 27 | | (GRPC,1) | init | 27 | | (GRPC,1) | MiniOzoneCluster | 423 | | (GRPC,1) | createRatisCluster | 434 | | (GRPC,1) | shutdown | 85 | | (GRPC,1) | sleep | 67 | | (local,1) | init | 67 | | (local,1) | MiniOzoneCluster | 463 | | (local,1) | shutdown | 115 | | (local,1) | sleep | 106 | | (NETTY,1) | init | 106 | | (NETTY,1) | MiniOzoneCluster | 507 | | (NETTY,1) | createRatisCluster | 530 | | (NETTY,1) | shutdown | 182 | | (NETTY,1) | sleep | 166 | As shown in the table, some threads are not terminated after MiniOzoneCluster shutdown and a 10-second sleep. For example, in (distributed,1), it has 6 threads initially but there are 27 threads after sleep. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11246) FSNameSystem#logAuditEvent should be called outside the read or write locks
[
https://issues.apache.org/jira/browse/HDFS-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093843#comment-16093843
]
Daryn Sharp commented on HDFS-11246:
I'm not sure I like either of the the suggestions because it re-orders the
audit logging and edit log syncing. On a semantic level, one might expect the
audit log to only include durable edits. It may be confusing to see audits for
edits lost in a crash.
More importantly logging before syncing is likely to increase the odds of less
txns batched per sync. Log4j synchronization can be a real bottleneck, even
with the async appender support I added.
The double try feels a bit odd but it seems like a clean change that doesn't
cause edit/audit reordering.
> FSNameSystem#logAuditEvent should be called outside the read or write locks
> ---
>
> Key: HDFS-11246
> URL: https://issues.apache.org/jira/browse/HDFS-11246
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-11246.001.patch, HDFS-11246.002.patch
>
>
> {code}
> readLock();
> boolean success = true;
> ContentSummary cs;
> try {
> checkOperation(OperationCategory.READ);
> cs = FSDirStatAndListingOp.getContentSummary(dir, src);
> } catch (AccessControlException ace) {
> success = false;
> logAuditEvent(success, operationName, src);
> throw ace;
> } finally {
> readUnlock(operationName);
> }
> {code}
> It would be nice to have audit logging outside the lock esp. in scenarios
> where applications hammer a given operation several times.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12164) CodecUtil throws stacktrace to the client in createRawEncoder/Decoder warn msg
[
https://issues.apache.org/jira/browse/HDFS-12164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HDFS-12164:
--
Summary: CodecUtil throws stacktrace to the client in
createRawEncoder/Decoder warn msg (was: CodecUtil throws stacktrace to the
client on createRawEncoder/Decoder error)
> CodecUtil throws stacktrace to the client in createRawEncoder/Decoder warn msg
> --
>
> Key: HDFS-12164
> URL: https://issues.apache.org/jira/browse/HDFS-12164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>
> While doing filesystem operations on ec dirs/files, the _failure to create
> raw encoder/decoder_ warning at the client side is accompanied by the
> stacktrace of the error.
> {code}
> 2017-07-19 21:31:00,209 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2017-07-19 21:31:01,023 WARN erasurecode.CodecUtil: Failed to create raw
> erasure encoder rs_native, fallback to next codec if possible
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory.createEncoder(NativeRSRawErasureCoderFactory.java:35)
> at
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:177)
> at
> org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:129)
> at
> org.apache.hadoop.hdfs.DFSStripedOutputStream.(DFSStripedOutputStream.java:302)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:309)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1216)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1133)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:451)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:389)
> at
> org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:181)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1074)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1054)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:943)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:509)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:484)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
> at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
> at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
> at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
> at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
> at
> org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
> at
> org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:286)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> Caused by: java.lang.RuntimeException: hadoop native library cannot be loaded.
> at
> org.apache.hadoop.io.erasurecode.ErasureCodeNative.checkNativeCodeLoaded(ErasureCodeNative.java:69)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder.(NativeRSRawEncoder.java:33)
> {code}
> We can just sh
[jira] [Created] (HDFS-12164) CodecUtil throws stacktrace to the client on createRawEncoder/Decoder error
Hanisha Koneru created HDFS-12164:
-
Summary: CodecUtil throws stacktrace to the client on
createRawEncoder/Decoder error
Key: HDFS-12164
URL: https://issues.apache.org/jira/browse/HDFS-12164
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding
Reporter: Hanisha Koneru
Assignee: Hanisha Koneru
While doing filesystem operations on ec dirs/files, the _failure to create raw
encoder/decoder_ warning at the client side is accompanied by the stacktrace of
the error.
{code}
2017-07-19 21:31:00,209 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2017-07-19 21:31:01,023 WARN erasurecode.CodecUtil: Failed to create raw
erasure encoder rs_native, fallback to next codec if possible
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory.createEncoder(NativeRSRawErasureCoderFactory.java:35)
at
org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoderWithFallback(CodecUtil.java:177)
at
org.apache.hadoop.io.erasurecode.CodecUtil.createRawEncoder(CodecUtil.java:129)
at
org.apache.hadoop.hdfs.DFSStripedOutputStream.(DFSStripedOutputStream.java:302)
at
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:309)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1216)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1133)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:451)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:448)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:462)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:389)
at
org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:181)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1074)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1054)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:943)
at
org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:509)
at
org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:484)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
at
org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
at
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:303)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:285)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:269)
at
org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
at
org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:286)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
Caused by: java.lang.RuntimeException: hadoop native library cannot be loaded.
at
org.apache.hadoop.io.erasurecode.ErasureCodeNative.checkNativeCodeLoaded(ErasureCodeNative.java:69)
at
org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawEncoder.(NativeRSRawEncoder.java:33)
{code}
We can just show the warning message to the client and avoid throwing the full
stacktrace.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-12163: --- Attachment: TestOzoneThreadCount20170719.patch TestOzoneThreadCount20170719.patch: the thread count test (TestOzoneThreadCount). > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-12163: --- Description: Checked the number of active threads used in MiniOzoneCluster with various settings: - Local handlers - Distributed handlers - Ratis-Netty - Ratis-gRPC The results are similar for all the settings. It uses 400+ threads for an 1-datanode MiniOzoneCluster. Moreover, there is a thread leak -- a number of the threads do not shutdown after the test is finished. Therefore, when tests run consecutively, the later tests use more threads. Will post the details in comments. was: Checked the number of active threads used in MiniOzoneCluster with various settings: - Local handlers - Distributed handlers - Ratis-Netty - Ratis-gRPC The results are similar for all the settings. It uses 400+ threads for a 1-datanode MiniOzoneCluster. Moreover, there is a thread leak -- a number of the threads do not shutdown after the test is finished. Therefore, when tests run consecutively, the later tests use more threads. Will post the details in comments. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-12163: --- Description: Checked the number of active threads used in MiniOzoneCluster with various settings: - Local handlers - Distributed handlers - Ratis-Netty - Ratis-gRPC The results are similar for all the settings. It uses 400+ threads for a 1-datanode MiniOzoneCluster. Moreover, there is a thread leak -- a number of the threads do not shutdown after the test is finished. Therefore, when tests run consecutively, the later tests use more threads. Will post the details in comments. was: Checked the number of active threads used in MiniOzoneCluster with various settings: - Local handlers - Distributed handlers - Ratis-Netty - Ratis-gRPC The results are similar for all the settings. It uses 400+ threads for a 1-datanode . Moreover, there is a thread leak -- a number of the threads do not shutdown after the test is finished. Therefore, when tests run consecutively, the later tests use more threads. Will post the details in comments. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for a > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-12163: --- Description: Checked the number of active threads used in MiniOzoneCluster with various settings: - Local handlers - Distributed handlers - Ratis-Netty - Ratis-gRPC The results are similar for all the settings. It uses 400+ threads for a 1-datanode . Moreover, there is a thread leak -- a number of the threads do not shutdown after the test is finished. Therefore, when tests run consecutively, the later tests use more threads. Will post the details in comments. was: Checked the number of active threads used in MiniOzoneCluster with various settings: - Local handlers - Distributed handlers - Ratis-Netty - Ratis-gRPC The results are similar for all the settings. It uses 400+ threads. Moreover, there is a thread leak -- a number of the threads do not shutdown after the test is finished. Therefore, when tests run consecutively, the later tests use more threads. Will post the details in comments. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for a > 1-datanode . > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
Tsz Wo Nicholas Sze created HDFS-12163: -- Summary: Ozone: MiniOzoneCluster uses 400+ threads Key: HDFS-12163 URL: https://issues.apache.org/jira/browse/HDFS-12163 Project: Hadoop HDFS Issue Type: Bug Components: ozone, test Reporter: Tsz Wo Nicholas Sze Checked the number of active threads used in MiniOzoneCluster with various settings: - Local handlers - Distributed handlers - Ratis-Netty - Ratis-gRPC The results are similar for all the settings. It uses 400+ threads. Moreover, there is a thread leak -- a number of the threads do not shutdown after the test is finished. Therefore, when tests run consecutively, the later tests use more threads. Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11246) FSNameSystem#logAuditEvent should be called outside the read or write locks
[
https://issues.apache.org/jira/browse/HDFS-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla updated HDFS-11246:
---
Summary: FSNameSystem#logAuditEvent should be called outside the read or
write locks (was: FSNameSystem#logAuditEvent should be called outside the read
or write locks during operations like getContentSummary)
> FSNameSystem#logAuditEvent should be called outside the read or write locks
> ---
>
> Key: HDFS-11246
> URL: https://issues.apache.org/jira/browse/HDFS-11246
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-11246.001.patch, HDFS-11246.002.patch
>
>
> {code}
> readLock();
> boolean success = true;
> ContentSummary cs;
> try {
> checkOperation(OperationCategory.READ);
> cs = FSDirStatAndListingOp.getContentSummary(dir, src);
> } catch (AccessControlException ace) {
> success = false;
> logAuditEvent(success, operationName, src);
> throw ace;
> } finally {
> readUnlock(operationName);
> }
> {code}
> It would be nice to have audit logging outside the lock esp. in scenarios
> where applications hammer a given operation several times.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12160) Fix broken NameNode metrics documentation
[
https://issues.apache.org/jira/browse/HDFS-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-12160:
---
Resolution: Not A Problem
Status: Resolved (was: Patch Available)
> Fix broken NameNode metrics documentation
> -
>
> Key: HDFS-12160
> URL: https://issues.apache.org/jira/browse/HDFS-12160
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha4
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Trivial
> Attachments: HDFS-12160.000.patch
>
>
> HDFS-11261 introduced documentation in {{Metrics.md}} for the metrics added
> in HDFS-10872. The metrics have a pipe ({{|}}) in them which breaks the
> markdown table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12160) Fix broken NameNode metrics documentation
[
https://issues.apache.org/jira/browse/HDFS-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093776#comment-16093776
]
Erik Krogen commented on HDFS-12160:
Hey [~vagarychen], I think your markdown editor may have some issue with the
{code}...`*OperationName*`...{code} syntax but it's used throughout the rest of
the file without issue, e.g. see
{code}`rpcQueueTime`*num*`s50thPercentileLatency`{code} (you can see it outputs
fine at e.g.
[https://hadoop.apache.org/docs/r3.0.0-alpha4/hadoop-project-dist/hadoop-common/Metrics.html]).
However I actually just noticed that although _my_ markdown editor complains
about the {{(...|...)}} syntax within the table, it is used elsewhere within
the {{Metrics.md}} file without issue, e.g. see
{{RamDiskBlocksEvictionWindowsnums(50|75|90|95|99)thPercentileLatency}} on the
same linked metrics page. Going to go ahead and close this out since it's
clearly not an issue for the website.
> Fix broken NameNode metrics documentation
> -
>
> Key: HDFS-12160
> URL: https://issues.apache.org/jira/browse/HDFS-12160
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha4
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Trivial
> Attachments: HDFS-12160.000.patch
>
>
> HDFS-11261 introduced documentation in {{Metrics.md}} for the metrics added
> in HDFS-10872. The metrics have a pipe ({{|}}) in them which breaks the
> markdown table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11920) Ozone : add key partition
[ https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11920: -- Attachment: HDFS-11920-HDFS-7240.004.patch Rebased with v004 patch. > Ozone : add key partition > - > > Key: HDFS-11920 > URL: https://issues.apache.org/jira/browse/HDFS-11920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11920-HDFS-7240.001.patch, > HDFS-11920-HDFS-7240.002.patch, HDFS-11920-HDFS-7240.003.patch, > HDFS-11920-HDFS-7240.004.patch > > > Currently, each key corresponds to one single SCM block, and putKey/getKey > writes/reads to this single SCM block. This works fine for keys with > reasonably small data size. However if the data is too huge, (e.g. not even > fits into a single container), then we need to be able to partition the key > data into multiple blocks, each in one container. This JIRA changes the > key-related classes to support this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[
https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093623#comment-16093623
]
Hadoop QA commented on HDFS-10285:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 7s{color} | {color:orange} hadoop-hdfs-project: The patch generated 11 new
+ 1967 unchanged - 2 fixed = 1978 total (was 1969) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-10285 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877947/HDFS-10285-consolidated-merge-patch-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc xml |
| uname | Linux 9d1d05c58c49 3.13.0-123-generic #172-Ubuntu SMP
[jira] [Updated] (HDFS-12139) HTTPFS liststatus returns incorrect pathSuffix for path of file
[
https://issues.apache.org/jira/browse/HDFS-12139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-12139:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
2.9.0
Status: Resolved (was: Patch Available)
> HTTPFS liststatus returns incorrect pathSuffix for path of file
> ---
>
> Key: HDFS-12139
> URL: https://issues.apache.org/jira/browse/HDFS-12139
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: httpfs
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12139.001.patch, HDFS-12139.002.patch
>
>
> Per the following logs, we can see that liststatus returns the same
> pathSuffix "test.txt" for /tmp/yj/yj1 and /tmp/yj/yj1/test.txt, which is
> wrong. The pathSuffix for the latter should be empty.
> {code}
> [thost ~]$ hadoop fs -copyFromLocal test.txt /tmp/yj/yj1
> [thost ~]$ curl
> "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1?op=LISTSTATUS&user.name=tuser";
> {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":157684989,"modificationTime":157685286,"blockSize":134217728,"replication":3}]}}
> [thost ~]$ curl
> "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1/test.txt?op=LISTSTATUS&user.name=tuser";
> {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":157684989,"modificationTime":157685286,"blockSize":134217728,"replication":3}]}}
> {code}
> Due to this bug, the symptom of running distcp with httpfs to copy a single
> file /tmp/test.txt is:
> {quote}
> WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:systest (auth:PROXY) via httpfs (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=systest, access=EXECUTE,
> inode="/tmp/test.txt":systest:supergroup:-rw-r--r-- (Ancestor /tmp/test.txt
> is not a directory).
> {quote}
> because /tmp/test.txt is treated as /tmp/test.txt/test.txt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12139) HTTPFS liststatus returns incorrect pathSuffix for path of file
[
https://issues.apache.org/jira/browse/HDFS-12139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093621#comment-16093621
]
Yongjun Zhang commented on HDFS-12139:
--
Thanks a lot [~xiaochen], committed to trunk and branch-2. I also created
HDFS-12162 per our discussion.
> HTTPFS liststatus returns incorrect pathSuffix for path of file
> ---
>
> Key: HDFS-12139
> URL: https://issues.apache.org/jira/browse/HDFS-12139
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: httpfs
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12139.001.patch, HDFS-12139.002.patch
>
>
> Per the following logs, we can see that liststatus returns the same
> pathSuffix "test.txt" for /tmp/yj/yj1 and /tmp/yj/yj1/test.txt, which is
> wrong. The pathSuffix for the latter should be empty.
> {code}
> [thost ~]$ hadoop fs -copyFromLocal test.txt /tmp/yj/yj1
> [thost ~]$ curl
> "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1?op=LISTSTATUS&user.name=tuser";
> {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":157684989,"modificationTime":157685286,"blockSize":134217728,"replication":3}]}}
> [thost ~]$ curl
> "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1/test.txt?op=LISTSTATUS&user.name=tuser";
> {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":157684989,"modificationTime":157685286,"blockSize":134217728,"replication":3}]}}
> {code}
> Due to this bug, the symptom of running distcp with httpfs to copy a single
> file /tmp/test.txt is:
> {quote}
> WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:systest (auth:PROXY) via httpfs (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=systest, access=EXECUTE,
> inode="/tmp/test.txt":systest:supergroup:-rw-r--r-- (Ancestor /tmp/test.txt
> is not a directory).
> {quote}
> because /tmp/test.txt is treated as /tmp/test.txt/test.txt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12162) Update listStatus document to describe the behavior when the argument is a file
[ https://issues.apache.org/jira/browse/HDFS-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-12162: - Description: The listStatus method can take in either directory path or file path as input, however, currently both the javadoc and external document describe it as only taking directory as input. This jira is to update the document about the behavior when the argument is a file path. Thanks [~xiaochen] for the review and discussion in HDFS-12139, creating this jira is the result of our discussion there. > Update listStatus document to describe the behavior when the argument is a > file > --- > > Key: HDFS-12162 > URL: https://issues.apache.org/jira/browse/HDFS-12162 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, httpfs >Reporter: Yongjun Zhang > > The listStatus method can take in either directory path or file path as > input, however, currently both the javadoc and external document describe it > as only taking directory as input. This jira is to update the document about > the behavior when the argument is a file path. > Thanks [~xiaochen] for the review and discussion in HDFS-12139, creating this > jira is the result of our discussion there. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12139) HTTPFS liststatus returns incorrect pathSuffix for path of file
[
https://issues.apache.org/jira/browse/HDFS-12139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093616#comment-16093616
]
Hudson commented on HDFS-12139:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12034 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12034/])
HDFS-12139. HTTPFS liststatus returns incorrect pathSuffix for path of (yzhang:
rev 3556e36be30211f46ac38899ce11a4d4cac6d635)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/test/java/org/apache/hadoop/fs/http/client/BaseTestHttpFSWith.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
> HTTPFS liststatus returns incorrect pathSuffix for path of file
> ---
>
> Key: HDFS-12139
> URL: https://issues.apache.org/jira/browse/HDFS-12139
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: httpfs
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-12139.001.patch, HDFS-12139.002.patch
>
>
> Per the following logs, we can see that liststatus returns the same
> pathSuffix "test.txt" for /tmp/yj/yj1 and /tmp/yj/yj1/test.txt, which is
> wrong. The pathSuffix for the latter should be empty.
> {code}
> [thost ~]$ hadoop fs -copyFromLocal test.txt /tmp/yj/yj1
> [thost ~]$ curl
> "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1?op=LISTSTATUS&user.name=tuser";
> {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":157684989,"modificationTime":157685286,"blockSize":134217728,"replication":3}]}}
> [thost ~]$ curl
> "http://thost.x.y:14000/webhdfs/v1/tmp/yj/yj1/test.txt?op=LISTSTATUS&user.name=tuser";
> {"FileStatuses":{"FileStatus":[{"pathSuffix":"test.txt","type":"FILE","length":16,"owner":"tuser","group":"supergroup","permission":"644","accessTime":157684989,"modificationTime":157685286,"blockSize":134217728,"replication":3}]}}
> {code}
> Due to this bug, the symptom of running distcp with httpfs to copy a single
> file /tmp/test.txt is:
> {quote}
> WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:systest (auth:PROXY) via httpfs (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=systest, access=EXECUTE,
> inode="/tmp/test.txt":systest:supergroup:-rw-r--r-- (Ancestor /tmp/test.txt
> is not a directory).
> {quote}
> because /tmp/test.txt is treated as /tmp/test.txt/test.txt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12162) Update listStatus document to describe the behavior when the argument is a file
Yongjun Zhang created HDFS-12162: Summary: Update listStatus document to describe the behavior when the argument is a file Key: HDFS-12162 URL: https://issues.apache.org/jira/browse/HDFS-12162 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs, httpfs Reporter: Yongjun Zhang -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12160) Fix broken NameNode metrics documentation
[
https://issues.apache.org/jira/browse/HDFS-12160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093607#comment-16093607
]
Chen Liang commented on HDFS-12160:
---
Thanks for filing this [~xkrogen] for filing this! I tried the patch and opened
in a markdown editor, I'm seeing something like
{code}
FSN{Read,Write}Lock OperationName NumOps | Total number of acquiring lock by
operations
{code}
On the doc page, similarly the other line. Is it what it is meant to be? Why
not just {{NumOps}} for the first field here?
> Fix broken NameNode metrics documentation
> -
>
> Key: HDFS-12160
> URL: https://issues.apache.org/jira/browse/HDFS-12160
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha4
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Trivial
> Attachments: HDFS-12160.000.patch
>
>
> HDFS-11261 introduced documentation in {{Metrics.md}} for the metrics added
> in HDFS-10872. The metrics have a pipe ({{|}}) in them which breaks the
> markdown table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092423#comment-16092423
]
Konstantin Shvachko edited comment on HDFS-11896 at 7/19/17 6:43 PM:
-
I was thinking how to fix this. Think jira should fix (1) starting from trunk
down to 2.7.4.
For (2) there was an easier fix for HDFS-9034, just to swap 2 lines in
{{HeartbeatManager.register()}}, that is first do {{updateHeartbeatState()}},
then {{addDatanode()}}. The former will reset everything to 0 in DND, including
nonDfsUsed and the latter will be adding 0 then. This would avoid breaking
followup jiras. We should just add the swapping in branch-2.7 patch here.
was (Author: shv):
I was thinking how to fix this. Think jira should fix (1) starting from trunk
down to 2.7.4.
For (2) there was an easier fix for HDFS-9034, just to swap 2 lines in
{{HeartbeatManager.register()}}, that is first do {{updateHeartbeatState()}},
then {{addDatanode()}}. The former will reset everything to in DND, including
nonDfsUsed and the latter will be adding 0 then. This would avoid breaking
followup jiras. We should just add the swapping in branch-2.7 patch here.
> Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
> ---
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Labels: release-blocker
> Attachments: HDFS-11896-002.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11896) Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
[
https://issues.apache.org/jira/browse/HDFS-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093590#comment-16093590
]
Konstantin Shvachko commented on HDFS-11896:
Hey [~brahmareddy],
* in your trunk patch you correctly reset nonDfsUsed to 0, but not other
fields. I was thinking we could factor out the part of
{{updateHeartbeatState()}} that sets the "total" fields excluding the update
times and use it in {{resetBlocks()}}. That way we know all fields are updated
even if somebody adds new ones.
* In the 2.7 patch you follow the HDFS-9034 path introducing
{{updateDnStat()}}, which will cause test failures as HDFS-9034 did. I
recommend instead to swap lines in {{HeartbeatManager.register()}} without
modifying {{addDatanode()}}.
> Non-dfsUsed will be doubled on dead node re-registration in branch-2.7.
> ---
>
> Key: HDFS-11896
> URL: https://issues.apache.org/jira/browse/HDFS-11896
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Labels: release-blocker
> Attachments: HDFS-11896-002.patch, HDFS-11896-branch-2.7-001.patch,
> HDFS-11896-branch-2.7-002.patch, HDFS-11896.patch
>
>
> *Scenario:*
> i)Make you sure you've non-dfs data.
> ii) Stop Datanode
> iii) wait it becomes dead
> iv) now restart and check the non-dfs data
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12158) Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-12158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093528#comment-16093528
]
Hudson commented on HDFS-12158:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12033 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12033/])
HDFS-12158. Secondary Namenode's web interface lack configs for (aengineer: rev
413b23eb04eee24275257ab462133e0818f87449)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeHttpServerXFrame.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
> Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
> --
>
> Key: HDFS-12158
> URL: https://issues.apache.org/jira/browse/HDFS-12158
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12158.001.patch
>
>
> HDFS-10579 adds X-FRAME-OPTIONS protection to Namenode and Datanode.
> This is also needed for Secondary Namenode as well.
> *Seondary Namenode misses X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50090/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1083
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
> *Primary Namenode offers X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50070/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> X-FRAME-OPTIONS: SAMEORIGIN
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1079
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12158) Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-12158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDFS-12158:
Resolution: Fixed
Hadoop Flags: Reviewed
Target Version/s: 2.8.2
Status: Resolved (was: Patch Available)
[~msingh] Thanks for the contribution. I have committed this to trunk,
branch-2, and branch-2.8.2
> Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
> --
>
> Key: HDFS-12158
> URL: https://issues.apache.org/jira/browse/HDFS-12158
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12158.001.patch
>
>
> HDFS-10579 adds X-FRAME-OPTIONS protection to Namenode and Datanode.
> This is also needed for Secondary Namenode as well.
> *Seondary Namenode misses X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50090/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1083
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
> *Primary Namenode offers X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50070/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> X-FRAME-OPTIONS: SAMEORIGIN
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1079
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12158) Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-12158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093431#comment-16093431
]
Anu Engineer commented on HDFS-12158:
-
+1, I will commit this shortly.
> Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
> --
>
> Key: HDFS-12158
> URL: https://issues.apache.org/jira/browse/HDFS-12158
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12158.001.patch
>
>
> HDFS-10579 adds X-FRAME-OPTIONS protection to Namenode and Datanode.
> This is also needed for Secondary Namenode as well.
> *Seondary Namenode misses X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50090/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1083
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
> *Primary Namenode offers X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50070/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> X-FRAME-OPTIONS: SAMEORIGIN
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1079
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12133) Correct ContentSummaryComputationContext Logger class name.
[
https://issues.apache.org/jira/browse/HDFS-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093372#comment-16093372
]
Surendra Singh Lilhore commented on HDFS-12133:
---
Thanks [~brahmareddy] for review and commit
> Correct ContentSummaryComputationContext Logger class name.
> ---
>
> Key: HDFS-12133
> URL: https://issues.apache.org/jira/browse/HDFS-12133
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha4
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12133.001.patch
>
>
> Now it is {code}public static final Log LOG =
> LogFactory.getLog(INode.class){code}
> It should be {{ContentSummaryComputationContext.class}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12158) Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-12158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093362#comment-16093362
]
Hadoop QA commented on HDFS-12158:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
42s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12158 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877996/HDFS-12158.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8db621cbdc82 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2843c68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/20338/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20338/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20338/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20338/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S
[jira] [Commented] (HDFS-12133) Correct ContentSummaryComputationContext Logger class name.
[
https://issues.apache.org/jira/browse/HDFS-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093337#comment-16093337
]
Hudson commented on HDFS-12133:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12032 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12032/])
HDFS-12133. Correct ContentSummaryComputationContext Logger class name..
(brahma: rev 04ff412dabf3f6b9d884171c4140adbc636d5387)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
> Correct ContentSummaryComputationContext Logger class name.
> ---
>
> Key: HDFS-12133
> URL: https://issues.apache.org/jira/browse/HDFS-12133
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha4
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12133.001.patch
>
>
> Now it is {code}public static final Log LOG =
> LogFactory.getLog(INode.class){code}
> It should be {{ContentSummaryComputationContext.class}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11246) FSNameSystem#logAuditEvent should be called outside the read or write locks during operations like getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093317#comment-16093317
]
Kuhu Shukla commented on HDFS-11246:
Thanks [~brahmareddy] for the review! Will update patch with option-2 approach
soon.
> FSNameSystem#logAuditEvent should be called outside the read or write locks
> during operations like getContentSummary
>
>
> Key: HDFS-11246
> URL: https://issues.apache.org/jira/browse/HDFS-11246
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-11246.001.patch, HDFS-11246.002.patch
>
>
> {code}
> readLock();
> boolean success = true;
> ContentSummary cs;
> try {
> checkOperation(OperationCategory.READ);
> cs = FSDirStatAndListingOp.getContentSummary(dir, src);
> } catch (AccessControlException ace) {
> success = false;
> logAuditEvent(success, operationName, src);
> throw ace;
> } finally {
> readUnlock(operationName);
> }
> {code}
> It would be nice to have audit logging outside the lock esp. in scenarios
> where applications hammer a given operation several times.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093313#comment-16093313
]
Kuhu Shukla commented on HDFS-5040:
---
For {{refreshServiceAcl}}, {{refreshUserToGroupsMappings}},
{{refreshSuperUserGroupsConfiguration}}, {{refreshCallQueue}}, {{refresh}}, we
handle super user checks at the Protocol level by
{code}
@KerberosInfo(
serverPrincipal=CommonConfigurationKeys.HADOOP_SECURITY_SERVICE_USER_NAME_KEY)
{code}
I think therefore we don't need to cover these as part of this change.
Appreciate any comments on this, [~brahmareddy].
> Audit log for admin commands/ logging output of all DFS admin commands
> --
>
> Key: HDFS-5040
> URL: https://issues.apache.org/jira/browse/HDFS-5040
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Raghu C Doppalapudi
>Assignee: Kuhu Shukla
> Labels: BB2015-05-TBR
> Attachments: HDFS-5040.001.patch, HDFS-5040.patch, HDFS-5040.patch,
> HDFS-5040.patch
>
>
> enable audit log for all the admin commands/also provide ability to log all
> the admin commands in separate log file, at this point all the logging is
> displayed on the console.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12067) Correct dfsadmin commands usage message to reflects IPC port
[ https://issues.apache.org/jira/browse/HDFS-12067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093301#comment-16093301 ] Hudson commented on HDFS-12067: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12031 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12031/]) HDFS-12067. Correct dfsadmin commands usage message to reflects IPC (brahma: rev f8cd55fe33665faf2d1b14df231516fc891118fc) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java > Correct dfsadmin commands usage message to reflects IPC port > > > Key: HDFS-12067 > URL: https://issues.apache.org/jira/browse/HDFS-12067 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: steven-wugang >Assignee: steven-wugang > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS_12067.001.patch, HDFS_12067.002.patch, > HDFS_12067.003.patch, HDFS_12067.004.patch, HDFS_12067.005.patch, > HDFS_12067.006.patch, HDFS-12067.patch > > > When I use the command,I see the command help description,but the help > description doesn't make it clear,especially the argument 'port',It's easy to > mistake for port (default 9866) in 'dfs.datanode.address'.Therefore, in order > to use this command better,I add some descriptions about the arguments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12133) Correct ContentSummaryComputationContext Logger class name.
[
https://issues.apache.org/jira/browse/HDFS-12133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12133:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
Status: Resolved (was: Patch Available)
Committed to trunk.[~surendrasingh] thanks for contribution and thanks to
[~vagarychen] for additional review.
> Correct ContentSummaryComputationContext Logger class name.
> ---
>
> Key: HDFS-12133
> URL: https://issues.apache.org/jira/browse/HDFS-12133
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha4
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HDFS-12133.001.patch
>
>
> Now it is {code}public static final Log LOG =
> LogFactory.getLog(INode.class){code}
> It should be {{ContentSummaryComputationContext.class}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12067) Correct dfsadmin commands usage message to reflects IPC port
[ https://issues.apache.org/jira/browse/HDFS-12067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12067: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) committed to trunk and branch-2. [~steven-wugang] thanks for contribution. and thanks to [~vagarychen] for additional review. > Correct dfsadmin commands usage message to reflects IPC port > > > Key: HDFS-12067 > URL: https://issues.apache.org/jira/browse/HDFS-12067 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: steven-wugang >Assignee: steven-wugang > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS_12067.001.patch, HDFS_12067.002.patch, > HDFS_12067.003.patch, HDFS_12067.004.patch, HDFS_12067.005.patch, > HDFS_12067.006.patch, HDFS-12067.patch > > > When I use the command,I see the command help description,but the help > description doesn't make it clear,especially the argument 'port',It's easy to > mistake for port (default 9866) in 'dfs.datanode.address'.Therefore, in order > to use this command better,I add some descriptions about the arguments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11886) Ozone : Optimize putKey operation to be async and consensus
[ https://issues.apache.org/jira/browse/HDFS-11886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-11886: --- Summary: Ozone : Optimize putKey operation to be async and consensus (was: Ozone : improve error handling for putkey operation) > Ozone : Optimize putKey operation to be async and consensus > --- > > Key: HDFS-11886 > URL: https://issues.apache.org/jira/browse/HDFS-11886 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang >Assignee: Weiwei Yang > Attachments: design-notes-putkey.pdf > > > Ozone's putKey operations involve a couple steps: > 1. KSM calls allocateBlock to SCM, writes this info to KSM's local metastore > 2. allocatedBlock gets returned to client, client checks to see if container > needs to be created on datanode, if yes, create the container > 3. writes the data to container. > it is possible that 1 succeeded, but 2 or 3 failed, in this case there will > be an entry in KSM's local metastore, but the key is actually nowhere to be > found. We need to revert 1 is 2 or 3 failed in this case. > To resolve this, we need at least two things to be implemented first. > 1. We need deleteKey() to be added KSM first. > 2. We also need container reports to be implemented first such that SCM can > track whether the container is actually added. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10874) libhdfs++: Public API headers should not depend on internal implementation
[ https://issues.apache.org/jira/browse/HDFS-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093208#comment-16093208 ] Stephen commented on HDFS-10874: Looks good. A couple of minor suggestions: * Try to avoid using pair<> for "first class types", such as in URI::query (also the plural, "queries", is more fitting) . It is easier to understand the meaning of "key" and "value" than "first" and "second". This is primarily a comment on the existing API but the changes in this JIRA make it more prominent. * parse_uri_into_optional was added to make tests backward compatible with previous API. Perhaps better to change the tests to use the updated API. > libhdfs++: Public API headers should not depend on internal implementation > -- > > Key: HDFS-10874 > URL: https://issues.apache.org/jira/browse/HDFS-10874 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10874.HDFS-8707.000.patch, > HDFS-10874.HDFS-8707.001.patch > > > Public headers need to do some combination of the following: stop including > parts of the implementation, forward declare bits of the implementation where > absolutely needed, or pull the implementation into include/hdfspp if it's > inseparable. > Example: > If you want to use the C++ API and only stick include/hdfspp in the include > path you'll get an error when you include include/hdfspp/options.h because > that goes and includes common/uri.h. > Related to the work described in HDFS-10787. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11886) Ozone : improve error handling for putkey operation
[ https://issues.apache.org/jira/browse/HDFS-11886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang reassigned HDFS-11886: -- Assignee: Weiwei Yang > Ozone : improve error handling for putkey operation > --- > > Key: HDFS-11886 > URL: https://issues.apache.org/jira/browse/HDFS-11886 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Chen Liang >Assignee: Weiwei Yang > Attachments: design-notes-putkey.pdf > > > Ozone's putKey operations involve a couple steps: > 1. KSM calls allocateBlock to SCM, writes this info to KSM's local metastore > 2. allocatedBlock gets returned to client, client checks to see if container > needs to be created on datanode, if yes, create the container > 3. writes the data to container. > it is possible that 1 succeeded, but 2 or 3 failed, in this case there will > be an entry in KSM's local metastore, but the key is actually nowhere to be > found. We need to revert 1 is 2 or 3 failed in this case. > To resolve this, we need at least two things to be implemented first. > 1. We need deleteKey() to be added KSM first. > 2. We also need container reports to be implemented first such that SCM can > track whether the container is actually added. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11936) Ozone: TestNodeManager times out before it is able to find all nodes
[
https://issues.apache.org/jira/browse/HDFS-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093192#comment-16093192
]
Weiwei Yang commented on HDFS-11936:
Hi [~yuanbo]
Thanks for working on this. I have some comment with this patch. It looks like
you narrow down the problem to {{SCMNodeManager#waitForHeartbeatProcessed}},
this check gives no guarantee that all heartbeats are processed. Can we fix
this method as it is also called some other places, in case they may have
similar problems. I am thinking if this can be fixed by
{code}
public boolean waitForHeartbeatProcessed() {
return this.heartbeatQueue.isEmpty();
}
{code}
Please let me know your thought. thanks.
> Ozone: TestNodeManager times out before it is able to find all nodes
>
>
> Key: HDFS-11936
> URL: https://issues.apache.org/jira/browse/HDFS-11936
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yuanbo Liu
> Attachments: HDFS-11936-HDFS-7240.001.patch
>
>
> During the pre-commit build of
> https://builds.apache.org/job/PreCommit-HDFS-Build/19795/testReport/
> we detected that a test in TestNodeManager is failing. Probably due to the
> fact that we need more time to execute this test in jenkins. This might be
> related to HDFS-11919
> The test failure report follows.
> ==
> {noformat}
> Regression
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport
> Failing for the past 1 build (Since Failed#19795 )
> Took 0.51 sec.
> Error Message
> expected:<2> but was:<18000>
> Stacktrace
> java.lang.AssertionError: expected:<2> but was:<18000>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport(TestNodeManager.java:972)
> Standard Output
> 2017-06-06 13:45:30,909 [main] INFO - Data node with ID:
> 732ebd32-a926-44c5-afbb-c9f87513a67c Registered.
> 2017-06-06 13:45:30,937 [main] INFO - Data node with ID:
> 6860fd5d-94dc-4ba8-acd0-41cc3fa7232d Registered.
> 2017-06-06 13:45:30,971 [main] INFO - Data node with ID:
> cad7174c-204c-4806-b3af-c874706d4bd9 Registered.
> 2017-06-06 13:45:30,996 [main] INFO - Data node with ID:
> 0130a672-719d-4b68-9a1e-13046f4281ff Registered.
> 2017-06-06 13:45:31,021 [main] INFO - Data node with ID:
> 8d9ea5d4-6752-48d4-9bf0-adb0bd1a651a Registered.
> 2017-06-06 13:45:31,046 [main] INFO - Data node with ID:
> f122e372-5a38-476b-97dc-5ae449190485 Registered.
> 2017-06-06 13:45:31,071 [main] INFO - Data node with ID:
> 5750eb03-c1ac-4b3a-bc59-c4d9481e245b Registered.
> 2017-06-06 13:45:31,097 [main] INFO - Data node with ID:
> aa2d90a1-9e85-41f8-a4e5-35c7d2ed7299 Registered.
> 2017-06-06 13:45:31,122 [main] INFO - Data node with ID:
> 5e52bf5c-7050-4fc9-bf10-0e52650229ee Registered.
> 2017-06-06 13:45:31,147 [main] INFO - Data node with ID:
> eaac7b8f-a556-4afc-9163-7309f7ccea18 Registered.
> 2017-06-06 13:45:31,224 [SCM Heartbeat Processing Thread - 0] INFO -
> Current Thread is interrupted, shutting down HB processing thread for Node
> Manager.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12158) Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-12158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12158:
-
Status: Patch Available (was: Open)
> Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
> --
>
> Key: HDFS-12158
> URL: https://issues.apache.org/jira/browse/HDFS-12158
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12158.001.patch
>
>
> HDFS-10579 adds X-FRAME-OPTIONS protection to Namenode and Datanode.
> This is also needed for Secondary Namenode as well.
> *Seondary Namenode misses X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50090/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1083
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
> *Primary Namenode offers X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50070/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> X-FRAME-OPTIONS: SAMEORIGIN
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1079
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12158) Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
[
https://issues.apache.org/jira/browse/HDFS-12158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDFS-12158:
-
Attachment: HDFS-12158.001.patch
> Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection
> --
>
> Key: HDFS-12158
> URL: https://issues.apache.org/jira/browse/HDFS-12158
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12158.001.patch
>
>
> HDFS-10579 adds X-FRAME-OPTIONS protection to Namenode and Datanode.
> This is also needed for Secondary Namenode as well.
> *Seondary Namenode misses X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50090/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:13:53 GMT
> Date: Tue, 18 Jul 2017 20:13:53 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1083
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
> *Primary Namenode offers X-FRAME-OPTIONS protection*
> {code}
> [root@f0e12b63907e opt]# curl -I http://127.0.0.1:50070/index.html
> HTTP/1.1 200 OK
> Cache-Control: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Expires: Tue, 18 Jul 2017 20:14:04 GMT
> Date: Tue, 18 Jul 2017 20:14:04 GMT
> Pragma: no-cache
> Content-Type: text/html; charset=utf-8
> X-FRAME-OPTIONS: SAMEORIGIN
> Last-Modified: Mon, 12 Jun 2017 13:15:41 GMT
> Content-Length: 1079
> Accept-Ranges: bytes
> Server: Jetty(6.1.26)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11246) FSNameSystem#logAuditEvent should be called outside the read or write locks during operations like getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093132#comment-16093132
]
Brahma Reddy Battula edited comment on HDFS-11246 at 7/19/17 2:09 PM:
--
[~kshukla] thanks for working on this.
bq.That would mean we log for IOExceptions and others as well with
allowed=false, which should ideally only be when we see an
AccessControlException. This was a bug fixed thru HDFS-9395. We can still have
the call in the finally block while making sure we don't log unconditionally.
Instead of double try-finally blocks can we log in finally block based on
some boolean condition like below such that it can be logged only on ACE..?
*Option 1:*
{code}
final String operationName = "setOwner";
-FileStatus auditStat;
+boolean logAudit = false;
+boolean allow = true;
+FileStatus auditStat=null;
checkOperation(OperationCategory.WRITE);
writeLock();
try {
checkOperation(OperationCategory.WRITE);
checkNameNodeSafeMode("Cannot set owner for " + src);
auditStat = FSDirAttrOp.setOwner(dir, src, username, group);
+ logAudit = true;
} catch (AccessControlException e) {
- logAuditEvent(false, operationName, src);
+ logAudit = true;
+ allow = false;
+
throw e;
} finally {
writeUnlock(operationName);
+ if(logAudit)
+logAuditEvent(allow, operationName, src, null, auditStat);
}
getEditLog().logSync();
-logAuditEvent(true, operationName, src, null, auditStat);
{code}
*Option 2:*
{code}
+ public enum AuditCheck {
+SUCCESS, FAILURE, NONE
+ }
/**
* Set owner for an existing file.
* @throws IOException
@@ -1807,21 +1810,25 @@ void setPermission(String src, FsPermission permission)
throws IOException {
void setOwner(String src, String username, String group)
throws IOException {
final String operationName = "setOwner";
-FileStatus auditStat;
+AuditCheck auditCheck = AuditCheck.NONE;
+FileStatus auditStat = null;
checkOperation(OperationCategory.WRITE);
writeLock();
try {
checkOperation(OperationCategory.WRITE);
checkNameNodeSafeMode("Cannot set owner for " + src);
auditStat = FSDirAttrOp.setOwner(dir, src, username, group);
+ auditCheck = AuditCheck.SUCCESS;
} catch (AccessControlException e) {
- logAuditEvent(false, operationName, src);
+ auditCheck = AuditCheck.FAILURE;
throw e;
} finally {
writeUnlock(operationName);
+ if (auditCheck != AuditCheck.NONE)
+logAuditEvent(auditCheck == AuditCheck.SUCCESS, operationName, src,
+null, auditStat);
}
getEditLog().logSync();
-logAuditEvent(true, operationName, src, null, auditStat);
{code}
I prefer *option-2.*
was (Author: brahmareddy):
[~kshukla] thanks for working on this.
bq.That would mean we log for IOExceptions and others as well with
allowed=false, which should ideally only be when we see an
AccessControlException. This was a bug fixed thru HDFS-9395. We can still have
the call in the finally block while making sure we don't log unconditionally.
Instead of double try-finally blocks can we log in finally block based on
some boolean condition like below such that it can be logged only on ACE..?
{code}
final String operationName = "setOwner";
-FileStatus auditStat;
+boolean logAudit = false;
+boolean allow = true;
+FileStatus auditStat=null;
checkOperation(OperationCategory.WRITE);
writeLock();
try {
checkOperation(OperationCategory.WRITE);
checkNameNodeSafeMode("Cannot set owner for " + src);
auditStat = FSDirAttrOp.setOwner(dir, src, username, group);
+ logAudit = true;
} catch (AccessControlException e) {
- logAuditEvent(false, operationName, src);
+ logAudit = true;
+ allow = false;
+
throw e;
} finally {
writeUnlock(operationName);
+ if(logAudit)
+logAuditEvent(allow, operationName, src, null, auditStat);
}
getEditLog().logSync();
-logAuditEvent(true, operationName, src, null, auditStat);
{code}
{code}
> FSNameSystem#logAuditEvent should be called outside the read or write locks
> during operations like getContentSummary
>
>
> Key: HDFS-11246
> URL: https://issues.apache.org/jira/browse/HDFS-11246
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-11246.001.patch, HDFS-11246.002.patch
>
>
> {code}
> readLock();
> boolean success = true;
> ContentSummary cs;
> try {
> checkOperation(OperationCat
[jira] [Commented] (HDFS-11246) FSNameSystem#logAuditEvent should be called outside the read or write locks during operations like getContentSummary
[
https://issues.apache.org/jira/browse/HDFS-11246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093132#comment-16093132
]
Brahma Reddy Battula commented on HDFS-11246:
-
[~kshukla] thanks for working on this.
bq.That would mean we log for IOExceptions and others as well with
allowed=false, which should ideally only be when we see an
AccessControlException. This was a bug fixed thru HDFS-9395. We can still have
the call in the finally block while making sure we don't log unconditionally.
Instead of double try-finally blocks can we log in finally block based on
some boolean condition like below such that it can be logged only on ACE..?
{code}
final String operationName = "setOwner";
-FileStatus auditStat;
+boolean logAudit = false;
+boolean allow = true;
+FileStatus auditStat=null;
checkOperation(OperationCategory.WRITE);
writeLock();
try {
checkOperation(OperationCategory.WRITE);
checkNameNodeSafeMode("Cannot set owner for " + src);
auditStat = FSDirAttrOp.setOwner(dir, src, username, group);
+ logAudit = true;
} catch (AccessControlException e) {
- logAuditEvent(false, operationName, src);
+ logAudit = true;
+ allow = false;
+
throw e;
} finally {
writeUnlock(operationName);
+ if(logAudit)
+logAuditEvent(allow, operationName, src, null, auditStat);
}
getEditLog().logSync();
-logAuditEvent(true, operationName, src, null, auditStat);
{code}
{code}
> FSNameSystem#logAuditEvent should be called outside the read or write locks
> during operations like getContentSummary
>
>
> Key: HDFS-11246
> URL: https://issues.apache.org/jira/browse/HDFS-11246
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: HDFS-11246.001.patch, HDFS-11246.002.patch
>
>
> {code}
> readLock();
> boolean success = true;
> ContentSummary cs;
> try {
> checkOperation(OperationCategory.READ);
> cs = FSDirStatAndListingOp.getContentSummary(dir, src);
> } catch (AccessControlException ace) {
> success = false;
> logAuditEvent(success, operationName, src);
> throw ace;
> } finally {
> readUnlock(operationName);
> }
> {code}
> It would be nice to have audit logging outside the lock esp. in scenarios
> where applications hammer a given operation several times.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093102#comment-16093102 ] Kuhu Shukla commented on HDFS-5040: --- Thank you so much [~brahmareddy] for the detailed review/comments. bq. I feel those also can be handled as part of this jira and DN commands we might need discuss further and handle in seperate jira..? Agreed. Will post an updated patch asap. bq. Hope you'll add testcases. Yes, I am working on it right now and wanted to get some comments on the approach through this patch. Will update patch soon. > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0-alpha1 >Reporter: Raghu C Doppalapudi >Assignee: Kuhu Shukla > Labels: BB2015-05-TBR > Attachments: HDFS-5040.001.patch, HDFS-5040.patch, HDFS-5040.patch, > HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[
https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092934#comment-16092934
]
Hadoop QA commented on HDFS-10285:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
22s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
42s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 6s{color} | {color:orange} hadoop-hdfs-project: The patch generated 11 new
+ 1967 unchanged - 2 fixed = 1978 total (was 1969) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-10285 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877947/HDFS-10285-consolidated-merge-patch-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit f
[jira] [Commented] (HDFS-11936) Ozone: TestNodeManager times out before it is able to find all nodes
[
https://issues.apache.org/jira/browse/HDFS-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092838#comment-16092838
]
Hadoop QA commented on HDFS-11936:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 9s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.ozone.container.replication.TestContainerReplicationManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11936 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12877930/HDFS-11936-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 858f7d247980 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / a715f60 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20336/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20336/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20336/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: TestNodeManager times out before it is able to find all nodes
> --
[jira] [Updated] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-10285: Attachment: HDFS-10285-consolidated-merge-patch-01.patch Attached new consolidated patch after fixing the checkstyle and test case failures in the branch. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11922) Ozone: KSM: Garbage collect deleted blocks
[ https://issues.apache.org/jira/browse/HDFS-11922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang reassigned HDFS-11922: -- Assignee: Weiwei Yang > Ozone: KSM: Garbage collect deleted blocks > -- > > Key: HDFS-11922 > URL: https://issues.apache.org/jira/browse/HDFS-11922 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Weiwei Yang >Priority: Critical > > We need to garbage collect deleted blocks from the Datanodes. There are two > cases where we will have orphaned blocks. One is like the classical HDFS, > where someone deletes a key and we need to delete the corresponding blocks. > Another case, is when someone overwrites a key -- an overwrite can be treated > as a delete and a new put -- that means that older blocks need to be GC-ed at > some point of time. > Couple of JIRAs has discussed this in one form or another -- so consolidating > all those discussions in this JIRA. > HDFS-11796 -- needs to fix this issue for some tests to pass > HDFS-11780 -- changed the old overwriting behavior to not supporting this > feature for time being. > HDFS-11920 - Once again runs into this issue when user tries to put an > existing key. > HDFS-11781 - delete key API in KSM only deletes the metadata -- and relies on > GC for Datanodes. > When we solve this issue, we should also consider 2 more aspects. > One, we support versioning in the buckets, tracking which blocks are really > orphaned is something that KSM will do. So delete and overwrite at some point > needs to decide how to handle versioning of buckets. > Two, If a key exists in a closed container, then it is immutable, hence the > strategy of removing the key might be more complex than just talking to an > open container. > cc : [~xyao], [~cheersyang], [~vagarychen], [~msingh], [~yuanbo], > [~szetszwo], [~nandakumar131] > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12117) HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface
[
https://issues.apache.org/jira/browse/HDFS-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092768#comment-16092768
]
Wellington Chevreuil commented on HDFS-12117:
-
Reviewed the failed junit test, I don't think it's related, as it touched
different parts of the code and it's passing on my local build:
{noformat}
java.lang.AssertionError: array lengths differed, expected.length=2
actual.length=3
at org.junit.Assert.fail(Assert.java:88)
at
org.junit.internal.ComparisonCriteria.assertArraysAreSameLength(ComparisonCriteria.java:71)
at
org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:32)
at org.junit.Assert.internalArrayEquals(Assert.java:473)
at org.junit.Assert.assertArrayEquals(Assert.java:265)
at org.junit.Assert.assertArrayEquals(Assert.java:280)
at
org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.verifyBlockLocations(BaseTestHttpFSWith.java:1229)
at
org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.testGetFileBlockLocations(BaseTestHttpFSWith.java:1208)
at
org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.operation(BaseTestHttpFSWith.java:1106)
at
org.apache.hadoop.fs.http.client.BaseTestHttpFSWith.testOperation(BaseTestHttpFSWith.java:1134)
{noformat}
> HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST
> Interface
> ---
>
> Key: HDFS-12117
> URL: https://issues.apache.org/jira/browse/HDFS-12117
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: httpfs
>Affects Versions: 3.0.0-alpha3
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
> Attachments: HDFS-12117.003.patch, HDFS-12117.004.patch,
> HDFS-12117.005.patch, HDFS-12117.patch.01, HDFS-12117.patch.02
>
>
> Currently, HttpFS is lacking implementation for SNAPSHOT related methods from
> WebHDFS REST interface as defined by WebHDFS documentation [WebHDFS
> documentation|https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Snapshot_Operations]
> I would like to work on this implementation, following the existing design
> approach already implemented by other WebHDFS methods on current HttpFS
> project, so I'll be proposing an initial patch soon for reviews.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12152) [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve thread cleanup time
[
https://issues.apache.org/jira/browse/HDFS-12152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-12152:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
I have just pushed it to branch!
> [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve
> thread cleanup time
> -
>
> Key: HDFS-12152
> URL: https://issues.apache.org/jira/browse/HDFS-12152
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Minor
> Fix For: HDFS-10285
>
> Attachments: HDFS-12152-HDFS-10285-01.patch,
> HDFS-12152-HDFS-10285-02.patch, HDFS-12152-HDFS-10285-03.patch
>
>
> This jira to improve the StoragePolicySatisfyWorker#stop sequence of steps to
> improve the thread interruption and graceful shutdown quickly.
> I have observed that
> [TestDataNodeUUID#testUUIDRegeneration|https://builds.apache.org/job/PreCommit-HDFS-Build/20271/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeUUID/testUUIDRegeneration/]
> test case is getting timed out frequently. When analyzing, it looks like the
> below function is always taking 3 secs waiting period. Probably, we could
> improve the thread interruption sequence so that the thread should finish
> #run method quickly.
> {code}
> StoragePolicySatisfyWorker.java
> void waitToFinishWorkerThread() {
> try {
> movementTrackerThread.join(3000);
> } catch (InterruptedException ignore) {
> // ignore
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12152) [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve thread cleanup time
[
https://issues.apache.org/jira/browse/HDFS-12152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092736#comment-16092736
]
Uma Maheswara Rao G commented on HDFS-12152:
Thank you for analyzing the test failure. This is a good catch [~rakeshr]
+1 committing it.
I looked at the test failures, they seems to be unrelated. Findbugs also not
related to this patch.
> [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve
> thread cleanup time
> -
>
> Key: HDFS-12152
> URL: https://issues.apache.org/jira/browse/HDFS-12152
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Minor
> Attachments: HDFS-12152-HDFS-10285-01.patch,
> HDFS-12152-HDFS-10285-02.patch, HDFS-12152-HDFS-10285-03.patch
>
>
> This jira to improve the StoragePolicySatisfyWorker#stop sequence of steps to
> improve the thread interruption and graceful shutdown quickly.
> I have observed that
> [TestDataNodeUUID#testUUIDRegeneration|https://builds.apache.org/job/PreCommit-HDFS-Build/20271/testReport/org.apache.hadoop.hdfs.server.datanode/TestDataNodeUUID/testUUIDRegeneration/]
> test case is getting timed out frequently. When analyzing, it looks like the
> below function is always taking 3 secs waiting period. Probably, we could
> improve the thread interruption sequence so that the thread should finish
> #run method quickly.
> {code}
> StoragePolicySatisfyWorker.java
> void waitToFinishWorkerThread() {
> try {
> movementTrackerThread.join(3000);
> } catch (InterruptedException ignore) {
> // ignore
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11936) Ozone: TestNodeManager times out before it is able to find all nodes
[
https://issues.apache.org/jira/browse/HDFS-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-11936:
--
Status: Patch Available (was: Open)
> Ozone: TestNodeManager times out before it is able to find all nodes
>
>
> Key: HDFS-11936
> URL: https://issues.apache.org/jira/browse/HDFS-11936
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yuanbo Liu
> Attachments: HDFS-11936-HDFS-7240.001.patch
>
>
> During the pre-commit build of
> https://builds.apache.org/job/PreCommit-HDFS-Build/19795/testReport/
> we detected that a test in TestNodeManager is failing. Probably due to the
> fact that we need more time to execute this test in jenkins. This might be
> related to HDFS-11919
> The test failure report follows.
> ==
> {noformat}
> Regression
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport
> Failing for the past 1 build (Since Failed#19795 )
> Took 0.51 sec.
> Error Message
> expected:<2> but was:<18000>
> Stacktrace
> java.lang.AssertionError: expected:<2> but was:<18000>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.scm.node.TestNodeManager.testScmStatsFromNodeReport(TestNodeManager.java:972)
> Standard Output
> 2017-06-06 13:45:30,909 [main] INFO - Data node with ID:
> 732ebd32-a926-44c5-afbb-c9f87513a67c Registered.
> 2017-06-06 13:45:30,937 [main] INFO - Data node with ID:
> 6860fd5d-94dc-4ba8-acd0-41cc3fa7232d Registered.
> 2017-06-06 13:45:30,971 [main] INFO - Data node with ID:
> cad7174c-204c-4806-b3af-c874706d4bd9 Registered.
> 2017-06-06 13:45:30,996 [main] INFO - Data node with ID:
> 0130a672-719d-4b68-9a1e-13046f4281ff Registered.
> 2017-06-06 13:45:31,021 [main] INFO - Data node with ID:
> 8d9ea5d4-6752-48d4-9bf0-adb0bd1a651a Registered.
> 2017-06-06 13:45:31,046 [main] INFO - Data node with ID:
> f122e372-5a38-476b-97dc-5ae449190485 Registered.
> 2017-06-06 13:45:31,071 [main] INFO - Data node with ID:
> 5750eb03-c1ac-4b3a-bc59-c4d9481e245b Registered.
> 2017-06-06 13:45:31,097 [main] INFO - Data node with ID:
> aa2d90a1-9e85-41f8-a4e5-35c7d2ed7299 Registered.
> 2017-06-06 13:45:31,122 [main] INFO - Data node with ID:
> 5e52bf5c-7050-4fc9-bf10-0e52650229ee Registered.
> 2017-06-06 13:45:31,147 [main] INFO - Data node with ID:
> eaac7b8f-a556-4afc-9163-7309f7ccea18 Registered.
> 2017-06-06 13:45:31,224 [SCM Heartbeat Processing Thread - 0] INFO -
> Current Thread is interrupted, shutting down HB processing thread for Node
> Manager.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[
https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092518#comment-16092518
]
Brahma Reddy Battula edited comment on HDFS-5040 at 7/19/17 7:00 AM:
-
*Comments for patch:*
# ) Instead of {{checkPermissionsAndLogAuditEvent}}, can we name it as
{{checkSuperuserPrivilege(String operationName)}} since this check will be for
super user..?
# ) looks you missed success audit log for {{finalizeUpgrade}},only updated the
failure case(ACE).
# ) a) {code}4342 String operationName = "safemode";{code}
can we have something like below,to know the exact action..?
String opName = action.toString().toLowerCase();
b) we might need to handle {{default:}} case also..? so better introduce one
boolean variable..?
# ) checkSuperuserPrivilege() just move out of the writelock.
{code}
6162 try {
6163checkOperation(OperationCategory.WRITE);
6164checkNameNodeSafeMode("Cannot allow snapshot for " + path);
6165checkSuperuserPrivilege();
6166FSDirSnapshotOp.allowSnapshot(dir, snapshotManager, path);
6167success = true;
6168 } finally {
6169writeUnlock(operationName);
6170 }
6171} catch (AccessControlException ace) {
6172 logAuditEvent(success, operationName, path, null, null);
6173 throw ace;
{code}
# ) Quota commands needs to handle for specific command.Like below.
{code}private String getQuotaCommand(long nsQuota, long dsQuota) {
if (nsQuota == HdfsConstants.QUOTA_RESET
&& dsQuota == HdfsConstants.QUOTA_DONT_SET) {
return "clearQuota";
} else if (nsQuota == HdfsConstants.QUOTA_DONT_SET
&& dsQuota == HdfsConstants.QUOTA_RESET) {
return "clearSpaceQuota";
} else if (dsQuota == HdfsConstants.QUOTA_DONT_SET) {
return "setQuota";
} else {
return "setSpaceQuota";
}
}{code}
# ) *Also Check for usages of {{checkSuperuserPrivilege()}}, wherever this is
called, add audit logs for that RPC if not already there.also move
checkSuperuserPrivilege() check even before obtaining any lock.*
# ) Hope you'll add testcases.
was (Author: brahmareddy):
*Comments for patch:*
# ) Instead of {{checkPermissionsAndLogAuditEvent}}, can we name it as
{{checkSuperuserPrivilege(String operationName)}} since this check will be for
super user..?
# ) looks you missed success audit log for {{finalizeUpgrade}},only updated the
failure case(ACE).
# ) a) {code}4342 String operationName = "safemode";{code}
can we have something like below,to know the exact action..?
String opName = action.toString().toLowerCase();
b) we might need to handle {{default:}} case also..? so better introduce one
boolean variable..?
# ) checkSuperuserPrivilege() just move out of the writelock.
{code}
6162 try {
6163checkOperation(OperationCategory.WRITE);
6164checkNameNodeSafeMode("Cannot allow snapshot for " + path);
6165checkSuperuserPrivilege();
6166FSDirSnapshotOp.allowSnapshot(dir, snapshotManager, path);
6167success = true;
6168 } finally {
6169writeUnlock(operationName);
6170 }
6171} catch (AccessControlException ace) {
6172 logAuditEvent(success, operationName, path, null, null);
6173 throw ace;
{code}
# ) Quota commands needs to handle for specific command.
# ) *Also Check for usages of {{checkSuperuserPrivilege()}}, wherever this is
called, add audit logs for that RPC if not already there.also move
checkSuperuserPrivilege() check even before obtaining any lock.*
# ) Hope you'll add testcases.
> Audit log for admin commands/ logging output of all DFS admin commands
> --
>
> Key: HDFS-5040
> URL: https://issues.apache.org/jira/browse/HDFS-5040
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Affects Versions: 3.0.0-alpha1
>Reporter: Raghu C Doppalapudi
>Assignee: Kuhu Shukla
> Labels: BB2015-05-TBR
> Attachments: HDFS-5040.001.patch, HDFS-5040.patch, HDFS-5040.patch,
> HDFS-5040.patch
>
>
> enable audit log for all the admin commands/also provide ability to log all
> the admin commands in separate log file, at this point all the logging is
> displayed on the console.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
