[jira] [Updated] (HDFS-15643) EC: Fix checksum computation in case of native encoders
[
https://issues.apache.org/jira/browse/HDFS-15643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15643:
---
Fix Version/s: 3.2.2
Thanks [~ayushtkn] and [~ahussein] for your works. Backport to branch-3.2.2.
> EC: Fix checksum computation in case of native encoders
> ---
>
> Key: HDFS-15643
> URL: https://issues.apache.org/jira/browse/HDFS-15643
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ahmed Hussein
>Assignee: Ayush Saxena
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1, 3.4.0, 3.1.5, 3.2.3
>
> Attachments: HDFS-15643-01.patch, Test-Fix-01.patch,
> TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery17.log,
> org.apache.hadoop.hdfs.TestFileChecksum-output.txt,
> org.apache.hadoop.hdfs.TestFileChecksum.txt
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> There are many failures in {{TestFileChecksumCompositeCrc}}. The test cases
> {{testStripedFileChecksumWithMissedDataBlocksRangeQueryXX}} fail. The
> following is a sample of the stack trace in two of them Query7 and Query8.
> {code:bash}
> org.apache.hadoop.fs.PathIOException: `/striped/stripedFileChecksum1': Fail
> to get block checksum for
> LocatedStripedBlock{BP-1812707539-172.17.0.3-1602771351154:blk_-9223372036854775792_1001;
> getBlockSize()=37748736; corrupt=false; offset=0;
> locs=[DatanodeInfoWithStorage[127.0.0.1:36687,DS-b00139f0-4f28-4870-8f72-b726bd339e23,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:36303,DS-49a3c58e-da4a-4256-b1f9-893e4003ec94,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:43975,DS-ac278858-b6c8-424f-9e20-58d718dabe31,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:37507,DS-17f9d8d8-f8d3-443b-8df7-29416a2f5cb0,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:36441,DS-7e9d19b5-6220-465f-b33e-f8ed0e60fb07,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:42555,DS-ce679f5e-19fe-45b0-a0cd-8d8bec2f4735,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:39093,DS-4a7f54bb-dd39-4b5b-8dee-31a1b565cd7f,DISK],
>
> DatanodeInfoWithStorage[127.0.0.1:41699,DS-e1f939f3-37e7-413e-a522-934243477d81,DISK]];
> indices=[1, 2, 3, 4, 5, 6, 7, 8]}
> at
> org.apache.hadoop.hdfs.FileChecksumHelper$StripedFileNonStripedChecksumComputer.checksumBlocks(FileChecksumHelper.java:640)
> at
> org.apache.hadoop.hdfs.FileChecksumHelper$FileChecksumComputer.compute(FileChecksumHelper.java:252)
> at
> org.apache.hadoop.hdfs.DFSClient.getFileChecksumInternal(DFSClient.java:1851)
> at
> org.apache.hadoop.hdfs.DFSClient.getFileChecksumWithCombineMode(DFSClient.java:1871)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$34.doCall(DistributedFileSystem.java:1902)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$34.doCall(DistributedFileSystem.java:1899)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1916)
> at
> org.apache.hadoop.hdfs.TestFileChecksum.getFileChecksum(TestFileChecksum.java:584)
> at
> org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery(TestFileChecksum.java:295)
> at
> org.apache.hadoop.hdfs.TestFileChecksum.testStripedFileChecksumWithMissedDataBlocksRangeQuery7(TestFileChecksum.java:377)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> {code:bash}
> Error Message
> `/striped/stripedFileChecksum1': Fail to get block checksum for
> LocatedStripedBlock{BP-1299291876-172.17.0.3-1602771356932:blk_-9223372036854775792_1001;
> getBlockSize()=37748736; corrupt=false; offset=0;
>
[jira] [Comment Edited] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227189#comment-17227189
]
Yiqun Lin edited comment on HDFS-14090 at 11/6/20, 6:58 AM:
Hi [~fengnanli], some minor comments from me:
1. I see here we introduce CONCURRENT_NS for concurrent call, why not acquire
permit to corresponding ns instead of?
2. Current description of setting hdfs-rbf-default.xml can describe more. At
least, we need to mention:
* The setting name for how to configure handler count for each ns, also
include CONCURRENT_NS ns.
* The sum of dedicated handler count should be less than value of
dfs.federation.router.handler.count
3. It would be better to add this improvement in HDFSRouterFederation.md.
Comment #2 and #3 can be addressed in a follow-up JIRA, :).
was (Author: linyiqun):
Hi [~fengnanli], some minor comments from me:
1. I see here we introduce CONCURRENT_NS for concurrent call, why not acquire
permit to corresponding ns instead of?
2. Current description of setting hdfs-rbf-default.xml can describe more. At
least, we need to mention:
* The setting name for how to configure handler count for each ns, also
include CONCURRENT_NS ns.
* The sum of dedicated handler count should be less than value of
dfs.federation.router.handler.count
3. It would be better to add this improvement in HDFSRouterFederation.md.
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508365=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508365
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 06/Nov/20 06:36
Start Date: 06/Nov/20 06:36
Worklog Time Spent: 10m
Work Description: maobaolong commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-722902960
@ferhui @Hexiaoqiao Thanks for the simplify suggestion, It make the code
more clear.
```java
private static boolean deleteAllowed(final INodesInPath iip) {
if (iip.length() < 1 || iip.getLastINode() == null) {
if (NameNode.stateChangeLog.isDebugEnabled()) {
NameNode.stateChangeLog.debug(
"DIR* FSDirectory.unprotectedDelete: failed to remove "
+ iip.getPath() + " because it does not exist");
}
return false;
} else if (iip.length() == 1) { // src is the root
NameNode.stateChangeLog.warn(
"DIR* FSDirectory.unprotectedDelete: failed to remove " +
iip.getPath() + " because the root is not allowed to be
deleted");
return false;
}
return true;
}
```
As the `deleteAllowed` return false means this call don't allowed to delete,
so I think it make sense to record `allowed = false` to auditlog.
And when `iip.length() < 1 || iip.getLastINode() == null` or `iip.length()
== 1` make `deleteAllowed` return `false`, so for the current deleteAllowed`
method implementation, ` iip.getLastINode() == null` should make record
`allowed = false` to auditlog.
After an overlook at the delete flow, `iip.getLastINode()` might not be
`null`.
I push a new commit to simplify the test code, Please take another look,
thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508365)
Time Spent: 3h 20m (was: 3h 10m)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227189#comment-17227189
]
Yiqun Lin commented on HDFS-14090:
--
Hi [~fengnanli], some minor comments from me:
1. I see here we introduce CONCURRENT_NS for concurrent call, why not acquire
permit to corresponding ns instead of?
2. Current description of setting hdfs-rbf-default.xml can describe more. At
least, we need to mention:
* The setting name for how to configure handler count for each ns, also
include CONCURRENT_NS ns.
* The sum of dedicated handler count should be less than value of
dfs.federation.router.handler.count
3. It would be better to add this improvement in HDFSRouterFederation.md.
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508360
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 06/Nov/20 06:17
Start Date: 06/Nov/20 06:17
Worklog Time Spent: 10m
Work Description: maobaolong commented on a change in pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#discussion_r518544296
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLoggerWithCommands.java
##
@@ -1205,6 +1205,18 @@ public void testGetDatanodeStorageReport() throws
Exception {
}
}
+ @Test
+ public void testDeleteRoot() throws Exception {
+Path srcDir = new Path("/");
+fileSys = DFSTestUtil.getFileSystemAs(user1, conf);
+boolean result = fileSys.delete(srcDir, true);
+fileSys.close();
+assertTrue(result == false);
Review comment:
Indeed, fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508360)
Time Spent: 3h 10m (was: 3h)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508346
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 06/Nov/20 03:39
Start Date: 06/Nov/20 03:39
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-722791873
@maobaolong Thanks for your quick response. IIUC, value of allowed entry in
audit log is decided if the request have permission to operate at the
beginning. For this case, it makes sense to set allowed=false when delete root
directory. I am not sure if it is reasonable that we also set allowed=false
when `iip.getLastINode() = null`. Or the condition `iip.getLastINode() == null`
could happen inner delete operation for some corner case? If not, the
improvement is OK for me. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508346)
Time Spent: 3h (was: 2h 50m)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508345
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 06/Nov/20 03:16
Start Date: 06/Nov/20 03:16
Worklog Time Spent: 10m
Work Description: ferhui commented on a change in pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#discussion_r518500077
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLoggerWithCommands.java
##
@@ -1205,6 +1205,18 @@ public void testGetDatanodeStorageReport() throws
Exception {
}
}
+ @Test
+ public void testDeleteRoot() throws Exception {
+Path srcDir = new Path("/");
+fileSys = DFSTestUtil.getFileSystemAs(user1, conf);
+boolean result = fileSys.delete(srcDir, true);
+fileSys.close();
+assertTrue(result == false);
Review comment:
As CI reports, it could be more simple here, assertFalse(result)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508345)
Time Spent: 2h 50m (was: 2h 40m)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15668) RBF: Fix RouterRPCMetrics annocation and document misplaced error
[
https://issues.apache.org/jira/browse/HDFS-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227128#comment-17227128
]
Hongbing Wang commented on HDFS-15668:
--
Both +{color:#172b4d}hadoop.security.TestLdapGroupsMapping{color}+
{color:#172b4d}and{color}
+{color:#172b4d}hadoop.hdfs.server.federation.router.TestRouterRpc{color}+
{color:#172b4d}tests pass in local.{color}
> RBF: Fix RouterRPCMetrics annocation and document misplaced error
> -
>
> Key: HDFS-15668
> URL: https://issues.apache.org/jira/browse/HDFS-15668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.2.0
>Reporter: Hongbing Wang
>Assignee: Hongbing Wang
>Priority: Minor
> Attachments: HDFS-15668.001.patch
>
>
> I found that the description of the two fields: +{{ProxyOpFailureStandby}}+
> and +{{ProxyOpFailureCommunicate}}+ in the
> [website|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Metrics.html#RouterRPCMetrics]
> may be misplaced.
> When I reviewed the code to see the meaning of the two fields, I found that
> their descriptions were indeed misplaced.
> _Origin code_：
> {code:java}
> @Metric("Number of operations to fail to reach NN")
> private MutableCounterLong proxyOpFailureStandby;
> @Metric("Number of operations to hit a standby NN")
> private MutableCounterLong proxyOpFailureCommunicate;
> {code}
> _They should be_:
> {code:java}
> @Metric("Number of operations to hit a standby NN")
> private MutableCounterLong proxyOpFailureStandby;
> @Metric("Number of operations to fail to reach NN")
> private MutableCounterLong proxyOpFailureCommunicate;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15624) Fix the SetQuotaByStorageTypeOp problem after updating hadoop
[ https://issues.apache.org/jira/browse/HDFS-15624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227101#comment-17227101 ] huangtianhua commented on HDFS-15624: - [~brahmareddy][~vinayakumarb][~ayushtkn] sorry to disturb, would you please to review the code https://github.com/apache/hadoop/pull/2377? Thanks very much. > Fix the SetQuotaByStorageTypeOp problem after updating hadoop > --- > > Key: HDFS-15624 > URL: https://issues.apache.org/jira/browse/HDFS-15624 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.4.0 >Reporter: YaYun Wang >Priority: Major > Labels: pull-request-available, release-blocker > Time Spent: 6.5h > Remaining Estimate: 0h > > HDFS-15025 adds a new storage Type NVDIMM, changes the ordinal() of the enum > of StorageType. And, setting the quota by storageType depends on the > ordinal(), therefore, it may cause the setting of quota to be invalid after > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15548) Allow configuring DISK/ARCHIVE storage types on same device mount
[ https://issues.apache.org/jira/browse/HDFS-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227065#comment-17227065 ] Jing Zhao commented on HDFS-15548: -- The current PR looks good to me. Do you also want to take another look, [~hexiaoqiao]? > Allow configuring DISK/ARCHIVE storage types on same device mount > - > > Key: HDFS-15548 > URL: https://issues.apache.org/jira/browse/HDFS-15548 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Labels: pull-request-available > Time Spent: 8.5h > Remaining Estimate: 0h > > We can allow configuring DISK/ARCHIVE storage types on the same device mount > on two separate directories. > Users should be able to configure the capacity for each. Also, the datanode > usage report should report stats correctly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227012#comment-17227012
]
Stephen O'Donnell commented on HDFS-15569:
--
I just thought it would be almost as easy to solve this for the general case of
multiple failed upgrade attempts, rather than assume there can only be one
failed directory to remove. You could have CM auto-restarting, or someone may
not increase the memory enough and need a further restart. However, but the
time the DN fails again, there is a good chance the current.tmp was already
removed, as it does not fail immediately on startup, but after running for some
time.
I am happy to go ahead with proposal here (renaming to current.tmp) as it will
solve the problem in most cases where a single restart would be enough.
> Speed up the Storage#doRecover during datanode rolling upgrade
> ---
>
> Key: HDFS-15569
> URL: https://issues.apache.org/jira/browse/HDFS-15569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hemanth Boyina
>Assignee: Hemanth Boyina
>Priority: Major
> Attachments: HDFS-15569.001.patch, HDFS-15569.002.patch,
> HDFS-15569.003.patch
>
>
> When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not
> having enough memory upgrade failed , Adjusted memory configurations and re
> upgraded datanode ,
> Now datanode upgrade has taken more time , on analyzing found that
> Storage#deleteDir has taken more time in RECOVER_UPGRADE state
> {code:java}
> "Thread-28" #270 daemon prio=5 os_prio=0 tid=0x7fed5a9b8000 nid=0x2b5c
> runnable [0x7fdcdad2a000]"Thread-28" #270 daemon prio=5 os_prio=0
> tid=0x7fed5a9b8000 nid=0x2b5c runnable [0x7fdcdad2a000]
> java.lang.Thread.State: RUNNABLE at java.io.UnixFileSystem.delete0(Native
> Method) at java.io.UnixFileSystem.delete(UnixFileSystem.java:265) at
> java.io.File.delete(File.java:1041) at
> org.apache.hadoop.fs.FileUtil.deleteImpl(FileUtil.java:229) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:270) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:153) at
> org.apache.hadoop.hdfs.server.common.Storage.deleteDir(Storage.java:1348) at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.doRecover(Storage.java:782)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:174)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:224)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:253)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:455)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:389)
> - locked <0x7fdf08ec7548> (a
> org.apache.hadoop.hdfs.server.datanode.DataStorage) at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:557)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1761)
> - locked <0x7fdf08ec7598> (a
> org.apache.hadoop.hdfs.server.datanode.DataNode) at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1697)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:822)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15569) Speed up the Storage#doRecover during datanode rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-15569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226918#comment-17226918
]
Wei-Chiu Chuang commented on HDFS-15569:
This is not directly related to Hadoop, but Cloudera Manager has a feature to
attempt restart a failed node multiple times. I am not sure if that behavior
also applies to upgrade. [~sodonnell] is that where your concern comes from?
> Speed up the Storage#doRecover during datanode rolling upgrade
> ---
>
> Key: HDFS-15569
> URL: https://issues.apache.org/jira/browse/HDFS-15569
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hemanth Boyina
>Assignee: Hemanth Boyina
>Priority: Major
> Attachments: HDFS-15569.001.patch, HDFS-15569.002.patch,
> HDFS-15569.003.patch
>
>
> When upgrading datanode from hadoop 2.7.2 to 3.1.1 , because of jvm not
> having enough memory upgrade failed , Adjusted memory configurations and re
> upgraded datanode ,
> Now datanode upgrade has taken more time , on analyzing found that
> Storage#deleteDir has taken more time in RECOVER_UPGRADE state
> {code:java}
> "Thread-28" #270 daemon prio=5 os_prio=0 tid=0x7fed5a9b8000 nid=0x2b5c
> runnable [0x7fdcdad2a000]"Thread-28" #270 daemon prio=5 os_prio=0
> tid=0x7fed5a9b8000 nid=0x2b5c runnable [0x7fdcdad2a000]
> java.lang.Thread.State: RUNNABLE at java.io.UnixFileSystem.delete0(Native
> Method) at java.io.UnixFileSystem.delete(UnixFileSystem.java:265) at
> java.io.File.delete(File.java:1041) at
> org.apache.hadoop.fs.FileUtil.deleteImpl(FileUtil.java:229) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:270) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDeleteContents(FileUtil.java:285) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:182) at
> org.apache.hadoop.fs.FileUtil.fullyDelete(FileUtil.java:153) at
> org.apache.hadoop.hdfs.server.common.Storage.deleteDir(Storage.java:1348) at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.doRecover(Storage.java:782)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadStorageDirectory(BlockPoolSliceStorage.java:174)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:224)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:253)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:455)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:389)
> - locked <0x7fdf08ec7548> (a
> org.apache.hadoop.hdfs.server.datanode.DataStorage) at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:557)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1761)
> - locked <0x7fdf08ec7598> (a
> org.apache.hadoop.hdfs.server.datanode.DataNode) at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1697)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:392)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:282)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:822)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes. {Static}
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226806#comment-17226806
]
Íñigo Goiri commented on HDFS-14090:
Thanks [~fengnanli] for [^HDFS-14090.023.patch].
The code looks good to me, the dynamic approach in HDFS-14750 should be trivial
to implement.
+1 from my side.
Have you tested this in some actual cluster? Any experience on how it works?
Anybody else available to take a look at this?
> RBF: Improved isolation for downstream name nodes. {Static}
> ---
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: CR Hota
>Assignee: Fengnan Li
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch,
> HDFS-14090.006.patch, HDFS-14090.007.patch, HDFS-14090.008.patch,
> HDFS-14090.009.patch, HDFS-14090.010.patch, HDFS-14090.011.patch,
> HDFS-14090.012.patch, HDFS-14090.013.patch, HDFS-14090.014.patch,
> HDFS-14090.015.patch, HDFS-14090.016.patch, HDFS-14090.017.patch,
> HDFS-14090.018.patch, HDFS-14090.019.patch, HDFS-14090.020.patch,
> HDFS-14090.021.patch, HDFS-14090.022.patch, HDFS-14090.023.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508151
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 05/Nov/20 15:12
Start Date: 05/Nov/20 15:12
Worklog Time Spent: 10m
Work Description: maobaolong commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-722439482
[ERROR] Errors:
[ERROR]
TestDFSInotifyEventInputStreamKerberized.testWithKerberizedCluster:116 ? IO
De...
[ERROR]
TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks:80 ?
TestTimedOut
[ERROR] TestBootstrapAliasmap.setup:56 ? Bind Problem binding to
[0.0.0.0:50200] java
[ERROR]
TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization:380 ?
TestTimedOut
The CI check unit error seems have no related to this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508151)
Time Spent: 2h 40m (was: 2.5h)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508150=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508150
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 05/Nov/20 15:08
Start Date: 05/Nov/20 15:08
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-722437580
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 28s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 1s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | spotbugs | 3m 5s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 3s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| -0 :warning: | checkstyle | 0m 43s |
[/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2437/3/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 182 unchanged
- 0 fixed = 183 total (was 182) |
| +1 :green_heart: | mvnsite | 1m 9s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no
whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 48s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | findbugs | 3m 3s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 95m 50s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2437/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 185m 4s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2437/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2437 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d1fcb12254df 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git
[jira] [Work logged] (HDFS-15659) Set dfs.namenode.redundancy.considerLoad to false in MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-15659?focusedWorklogId=508143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508143
]
ASF GitHub Bot logged work on HDFS-15659:
-
Author: ASF GitHub Bot
Created on: 05/Nov/20 14:31
Start Date: 05/Nov/20 14:31
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2435:
URL: https://github.com/apache/hadoop/pull/2435#issuecomment-722414255
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 7s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | pathlen | 0m 0s |
[/pathlen.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2435/2/artifact/out/pathlen.txt)
| The patch appears to contain 1 files with names longer than 240 |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 411 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 55s | | trunk passed |
| +1 :green_heart: | compile | 21m 23s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | compile | 18m 12s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | checkstyle | 4m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 11m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 50s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 9m 29s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 9m 51s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | spotbugs | 0m 44s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +0 :ok: | findbugs | 0m 26s | |
branch/hadoop-hdfs-project/hadoop-hdfs-native-client no findbugs output file
(findbugsXml.xml) |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 8m 51s | | the patch passed |
| +1 :green_heart: | compile | 23m 9s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javac | 23m 9s | | the patch passed |
| +1 :green_heart: | compile | 19m 5s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | javac | 19m 5s | | the patch passed |
| -0 :warning: | checkstyle | 4m 27s |
[/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2435/2/artifact/out/diff-checkstyle-root.txt)
| root: The patch generated 180 new + 6030 unchanged - 40 fixed = 6210 total
(was 6070) |
| +1 :green_heart: | mvnsite | 11m 54s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2435/2/artifact/out/whitespace-tabs.txt)
| The patch 2 line(s) with tabs. |
| +1 :green_heart: | shadedclient | 16m 43s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 9m 27s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 10m 23s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | findbugs | 0m 27s | |
hadoop-hdfs-project/hadoop-hdfs-native-client has no data from findbugs |
_ Other Tests _ |
| -1 :x: | unit | 121m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2435/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 8m 36s | | hadoop-hdfs-native-client in
the patch passed. |
| -1 :x: | unit | 10m 21s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2435/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| -1 :x: | unit | 109m 12s |
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508106=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508106
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 05/Nov/20 11:58
Start Date: 05/Nov/20 11:58
Worklog Time Spent: 10m
Work Description: maobaolong commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-72200
@ferhui @Hexiaoqiao Thank you for the interesting about this PR.
> There is another confused point. If iip.getLastINode() == null returns
true, will also print allowed=false after changes,
```java
private static boolean deleteAllowed(final INodesInPath iip) {
if (iip.length() < 1 || iip.getLastINode() == null) {
if (NameNode.stateChangeLog.isDebugEnabled()) {
NameNode.stateChangeLog.debug(
"DIR* FSDirectory.unprotectedDelete: failed to remove "
+ iip.getPath() + " because it does not exist");
}
return false;
} else if (iip.length() == 1) { // src is the root
NameNode.stateChangeLog.warn(
"DIR* FSDirectory.unprotectedDelete: failed to remove " +
iip.getPath() + " because the root is not allowed to be
deleted");
return false;
}
return true;
}
```
Through this method, I think ` iip.getLastINode() == null` will returns
false, and auditlog will record `allowed = false`. not sure to catch your point?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508106)
Time Spent: 2h 20m (was: 2h 10m)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508090
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 05/Nov/20 11:20
Start Date: 05/Nov/20 11:20
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-722315425
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch
appears to include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 25s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | compile | 1m 12s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 1s | | branch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 51s | | trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +0 :ok: | spotbugs | 3m 11s | | Used deprecated FindBugs config;
considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 8s | | trunk passed |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| -0 :warning: | checkstyle | 0m 40s |
[/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2437/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 182 unchanged
- 0 fixed = 183 total (was 182) |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| -1 :x: | whitespace | 0m 0s |
[/whitespace-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2437/2/artifact/out/whitespace-eol.txt)
| The patch has 1 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | shadedclient | 14m 12s | | patch has no errors
when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
| +1 :green_heart: | findbugs | 3m 54s | | the patch passed |
_ Other Tests _ |
| -1 :x: | unit | 145m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2437/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 236m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.tools.TestECAdmin |
| | hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForXAttr |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.TestDistributedFileSystemWithECFile |
| |
[jira] [Commented] (HDFS-15668) RBF: Fix RouterRPCMetrics annocation and document misplaced error
[
https://issues.apache.org/jira/browse/HDFS-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226648#comment-17226648
]
Hadoop QA commented on HDFS-15668:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
54s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | | {color:blue} markdownlint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | | {color:red} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
8s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
10s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
58s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
43s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 39s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
19s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
44s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
44s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
58s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
58s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
40s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
24m 12s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green}
[jira] [Updated] (HDFS-15669) RBF: Incorrect GetQuota caused by different implementation of HashSet
[
https://issues.apache.org/jira/browse/HDFS-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15669:
Summary: RBF: Incorrect GetQuota caused by different implementation of
HashSet (was: [HDFS] [RBF] Incorrect GetQuota caused by different
implementation of HashSet )
> RBF: Incorrect GetQuota caused by different implementation of HashSet
> --
>
> Key: HDFS-15669
> URL: https://issues.apache.org/jira/browse/HDFS-15669
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Janus Chow
>Priority: Major
>
> In the method of Quota#getQuotaUsage, the result could be different on
> different versions of Java.
> The trace route is as follows:
> {code:java}
> Quota#getQuotaUsage
> - Quota#getValidQuotaLocations
> - Quota#getQuotaRemoteLocations
> - RouterQuotaManager#getPaths{code}
> In RouterQuotaManager#getPaths, the path is stored in a _HashSet_. And in
> Quota#getValidQuotaLocations, the values in HashSet would be used to check if
> the paths are parent-child relation. So paths in HashSet could affect the
> result of getValidQuotaLocations as follows:
> {code:java}
> // Situation.1 Locations in HashSet
> [ns0->/testdir7, ns0->/testdir7/subdir, ns1->/testdir8]
> // Situation.1 getQuota results
> {ns0->/testdir7= 10 6 100 100 , ns1->/testdir8= 10 8 100 100}
> // Situation.2 Locations in HashSet
> [ns0->/testdir7/subdir, ns1->/testdir8, ns0->/testdir7]
> // Situation.2 getQuota results
> {ns0->/testdir7= 10 8 100 100 , ns0->/testdir7/subdir= 10 6 100 100 ,
> ns1->/testdir8= 10 8 100 100 }{code}
> Situation.1 and Situation.2 happen when the underlying implementation of
> HashSet is different, that is Situation.2 happens using Java-7, and
> Situation.2 happens after Java-8.
> This problem can be solved when we do sorting on the results of
> Quota#_getQuotaRemoteLocations_, but I'm not sure if we should do it.
> {code:java}
> /**
> * Get all quota remote locations across subclusters under given
> * federation path.
> * @param path Federation path.
> * @return List of quota remote locations.
> * @throws IOException
> */
> private List getQuotaRemoteLocations(String path)
> throws IOException {
> List locations = new LinkedList<>();
> RouterQuotaManager manager = this.router.getQuotaManager();
> if (manager != null) {
> Set childrenPaths = manager.getPaths(path);
> for (String childPath : childrenPaths) {
> locations.addAll(rpcServer.getLocationsForPath(childPath, true, false));
> }
> }
> ++ Collections.sort(locations);
> return locations;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508078
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 05/Nov/20 10:36
Start Date: 05/Nov/20 10:36
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-722292766
Thanks @maobaolong and @ferhui for your detailed information. The
improvement seems OK for root directory. There is another confused point. If
`iip.getLastINode() == null` returns true, will also print `allowed=false`
after changes, actually for this case is INode does not exist rather than not
allowed. Any thought? Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508078)
Time Spent: 2h (was: 1h 50m)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15669) [HDFS] [RBF] Incorrect GetQuota caused by different implementation of HashSet
[
https://issues.apache.org/jira/browse/HDFS-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Janus Chow updated HDFS-15669:
--
Description:
In the method of Quota#getQuotaUsage, the result could be different on
different versions of Java.
The trace route is as follows:
{code:java}
Quota#getQuotaUsage
- Quota#getValidQuotaLocations
- Quota#getQuotaRemoteLocations
- RouterQuotaManager#getPaths{code}
In RouterQuotaManager#getPaths, the path is stored in a _HashSet_. And in
Quota#getValidQuotaLocations, the values in HashSet would be used to check if
the paths are parent-child relation. So paths in HashSet could affect the
result of getValidQuotaLocations as follows:
{code:java}
// Situation.1 Locations in HashSet
[ns0->/testdir7, ns0->/testdir7/subdir, ns1->/testdir8]
// Situation.1 getQuota results
{ns0->/testdir7= 10 6 100 100 , ns1->/testdir8= 10 8 100 100}
// Situation.2 Locations in HashSet
[ns0->/testdir7/subdir, ns1->/testdir8, ns0->/testdir7]
// Situation.2 getQuota results
{ns0->/testdir7= 10 8 100 100 , ns0->/testdir7/subdir= 10 6 100 100 ,
ns1->/testdir8= 10 8 100 100 }{code}
Situation.1 and Situation.2 happen when the underlying implementation of
HashSet is different, that is Situation.2 happens using Java-7, and Situation.2
happens after Java-8.
This problem can be solved when we do sorting on the results of
Quota#_getQuotaRemoteLocations_, but I'm not sure if we should do it.
{code:java}
/**
* Get all quota remote locations across subclusters under given
* federation path.
* @param path Federation path.
* @return List of quota remote locations.
* @throws IOException
*/
private List getQuotaRemoteLocations(String path)
throws IOException {
List locations = new LinkedList<>();
RouterQuotaManager manager = this.router.getQuotaManager();
if (manager != null) {
Set childrenPaths = manager.getPaths(path);
for (String childPath : childrenPaths) {
locations.addAll(rpcServer.getLocationsForPath(childPath, true, false));
}
}
++ Collections.sort(locations);
return locations;
}
{code}
was:
In the method of Quota#getQuotaUsage, the result could be different on
different versions of Java.
The trace route is as follows:
{code:java}
Quota#getQuotaUsage
- Quota#getValidQuotaLocations
- Quota#getQuotaRemoteLocations
- RouterQuotaManager#getPaths{code}
In RouterQuotaManager#getPaths, the path is stored in a _HashSet_. And in
Quota#getValidQuotaLocations, the values in HashSet would be used to check if
the paths are parent-child relation. So paths in HashSet could affect the
result of getValidQuotaLocations as follows:
{code:java}
// Situation.1 Locations in HashSet
[ns0->/testdir7, ns0->/testdir7/subdir, ns1->/testdir8]
// Situation.1 getQuota results
{ns0->/testdir7= 10 6 100 100 , ns1->/testdir8= 10 8 100 100}
// Situation.2 Locations in HashSet
[ns0->/testdir7/subdir, ns1->/testdir8, ns0->/testdir7]
// Situation.2 getQuota results
{ns0->/testdir7= 10 8 100 100 , ns0->/testdir7/subdir= 10 6 100 100 ,
ns1->/testdir8= 10 8 100 100 }{code}
Situation.1 and Situation.2 happen when the underlying implementation of
HashSet is different, that is Situation.2 happens using Java-7, and Situation.2
happens after Java-8.
This problem can be solved when we do sorting on the results of
Quota#_getQuotaRemoteLocations_.
{code:java}
/**
* Get all quota remote locations across subclusters under given
* federation path.
* @param path Federation path.
* @return List of quota remote locations.
* @throws IOException
*/
private List getQuotaRemoteLocations(String path)
throws IOException {
List locations = new LinkedList<>();
RouterQuotaManager manager = this.router.getQuotaManager();
if (manager != null) {
Set childrenPaths = manager.getPaths(path);
for (String childPath : childrenPaths) {
locations.addAll(rpcServer.getLocationsForPath(childPath, true, false));
}
}
++ Collections.sort(locations);
return locations;
}
{code}
> [HDFS] [RBF] Incorrect GetQuota caused by different implementation of HashSet
> --
>
> Key: HDFS-15669
> URL: https://issues.apache.org/jira/browse/HDFS-15669
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Janus Chow
>Priority: Major
>
> In the method of Quota#getQuotaUsage, the result could be different on
> different versions of Java.
> The trace route is as follows:
> {code:java}
> Quota#getQuotaUsage
> - Quota#getValidQuotaLocations
> - Quota#getQuotaRemoteLocations
> - RouterQuotaManager#getPaths{code}
> In RouterQuotaManager#getPaths, the path is stored in a _HashSet_. And in
> Quota#getValidQuotaLocations, the values in HashSet would be used to check if
> the paths are parent-child relation. So paths in HashSet could affect the
>
[jira] [Work logged] (HDFS-15667) Audit log record the unexpected allowed result when delete called
[
https://issues.apache.org/jira/browse/HDFS-15667?focusedWorklogId=508044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-508044
]
ASF GitHub Bot logged work on HDFS-15667:
-
Author: ASF GitHub Bot
Created on: 05/Nov/20 09:11
Start Date: 05/Nov/20 09:11
Worklog Time Spent: 10m
Work Description: ferhui commented on pull request #2437:
URL: https://github.com/apache/hadoop/pull/2437#issuecomment-722246700
@maobaolong Thanks for explanation. Got it now.
@Hexiaoqiao Could you please take a look again?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 508044)
Time Spent: 1h 50m (was: 1h 40m)
> Audit log record the unexpected allowed result when delete called
> -
>
> Key: HDFS-15667
> URL: https://issues.apache.org/jira/browse/HDFS-15667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1, 3.4.0
>Reporter: Baolong Mao
>Assignee: Baolong Mao
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I met this issue if rm root directory, for remove non-root and non-empty
> directory, toRemovedBlocks isn't null, its toDeleteList size is 0.
> !screenshot-1.png!
> when will return null?
> Through this screenshot, we can find that if fileRemoved = -1, then
> toRemovedBlocks = null
> !screenshot-2.png!
> And when deleteAllowed(iip) return false, fileRemoved can be -1,
> {code:java}
> private static boolean deleteAllowed(final INodesInPath iip) {
> if (iip.length() < 1 || iip.getLastINode() == null) {
> if (NameNode.stateChangeLog.isDebugEnabled()) {
> NameNode.stateChangeLog.debug(
> "DIR* FSDirectory.unprotectedDelete: failed to remove "
> + iip.getPath() + " because it does not exist");
> }
> return false;
> } else if (iip.length() == 1) { // src is the root
> NameNode.stateChangeLog.warn(
> "DIR* FSDirectory.unprotectedDelete: failed to remove " +
> iip.getPath() + " because the root is not allowed to be
> deleted");
> return false;
> }
> return true;
> }
> {code}
> Through the code of deleteAllowed, we can find that when src is the root, it
> can return false.
> So without this PR, when I execute *bin/hdfs dfs -rm -r /*
> I find the confusing auditlog line like following
> 2020-11-05 14:32:53,420 INFO FSNamesystem.audit
> (FSNamesystem.java:logAuditMessage(8102)) - allowed=true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
