[jira] [Commented] (HDFS-15407) Hedged read will not work if a datanode slow for a long time
[ https://issues.apache.org/jira/browse/HDFS-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135343#comment-17135343 ] liuyanyu commented on HDFS-15407: - [~ayushtkn] Could you please take a look at this issue? > Hedged read will not work if a datanode slow for a long time > > > Key: HDFS-15407 > URL: https://issues.apache.org/jira/browse/HDFS-15407 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1, datanode >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > > I use cgroups to limit the datanode node IO to 1024Byte/s, use hedged read to > read the file, (where dfs.client.hedged.read.threadpool.size is set to 5, > dfs.client.hedged.read.threshold.millis is set to 500), the first 5 buffer > read timeout, switch other datenode nodes to read successfully. Then stuck > for a long time because of SocketTimeoutException. Log as follows > 2020-06-11 16:40:07,832 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:08,562 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:09,102 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:09,642 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:10,182 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:10,182 | INFO | main | Execution rejected, Executing in > current thread | DFSClient.java:3049 > 2020-06-11 16:40:10,219 | INFO | main | Execution rejected, Executing in > current thread | DFSClient.java:3049 > 2020-06-11 16:50:07,638 | WARN | hedgedRead-0 | I/O error constructing > remote block reader. | BlockReaderFactory.java:764 > java.net.SocketTimeoutException: 60 millis timeout while waiting for > channel to be ready for read. ch : java.nio.channels.SocketChannel[connected > local=/xx.xx.xx.113:62750 remote=/xx.xx.xx.28:25009] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:418) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:853) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:749) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379) > at > org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:661) > at > org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1063) > at > org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1035) > at > org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1031) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-06-11 16:50:07,638 | WARN | hedgedRead-0 | Connection failure: Failed > to connect to /xx.xx.xx.28:25009 for file /testhdfs/test2.jar for block > BP-1820384660-xx.xx.xx.74-1585533043013:blk_1082582662_8861386:java.net.SocketTimeoutException: > 60 millis timeout while waiting for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/xx.xx.xx.113:62750 > remote=/xx.xx.xx.28
[jira] [Commented] (HDFS-15407) Hedged read will not work if a datanode slow for a long time
[ https://issues.apache.org/jira/browse/HDFS-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133095#comment-17133095 ] liuyanyu commented on HDFS-15407: - One reason is hedged Read cancel the timeout request using future.cancel(false). This will not cancel the request immediately，in-progress tasks are allowed to complete. So those requests to slow datanode will occupy the thread pool. And other request can not be submit. The other reason is the slow datanode will not be remembered when read next buffer. So the read request will continue to choose slow datanode , timeout and choose another one. Finally the requests to slow datanode occupy the thread pool. Other requests can not be submit. > Hedged read will not work if a datanode slow for a long time > > > Key: HDFS-15407 > URL: https://issues.apache.org/jira/browse/HDFS-15407 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1, datanode >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > > I use cgroups to limit the datanode node IO to 1024Byte/s, use hedged read to > read the file, (where dfs.client.hedged.read.threadpool.size is set to 5, > dfs.client.hedged.read.threshold.millis is set to 500), the first 5 buffer > read timeout, switch other datenode nodes to read successfully. Then stuck > for a long time because of SocketTimeoutException. Log as follows > 2020-06-11 16:40:07,832 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:08,562 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:09,102 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:09,642 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:10,182 | INFO | main | Waited 500ms to read from > DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; > spawning hedged read | DFSInputStream.java:1188 > 2020-06-11 16:40:10,182 | INFO | main | Execution rejected, Executing in > current thread | DFSClient.java:3049 > 2020-06-11 16:40:10,219 | INFO | main | Execution rejected, Executing in > current thread | DFSClient.java:3049 > 2020-06-11 16:50:07,638 | WARN | hedgedRead-0 | I/O error constructing > remote block reader. | BlockReaderFactory.java:764 > java.net.SocketTimeoutException: 60 millis timeout while waiting for > channel to be ready for read. ch : java.nio.channels.SocketChannel[connected > local=/xx.xx.xx.113:62750 remote=/xx.xx.xx.28:25009] > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:418) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:853) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:749) > at > org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379) > at > org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:661) > at > org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1063) > at > org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1035) > at > org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1031) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP
[jira] [Created] (HDFS-15407) Hedged read will not work if a datanode slow for a long time
liuyanyu created HDFS-15407: --- Summary: Hedged read will not work if a datanode slow for a long time Key: HDFS-15407 URL: https://issues.apache.org/jira/browse/HDFS-15407 Project: Hadoop HDFS Issue Type: Bug Components: 3.1.1, datanode Affects Versions: 3.1.1 Reporter: liuyanyu Assignee: liuyanyu I use cgroups to limit the datanode node IO to 1024Byte/s, use hedged read to read the file, (where dfs.client.hedged.read.threadpool.size is set to 5, dfs.client.hedged.read.threshold.millis is set to 500), the first 5 buffer read timeout, switch other datenode nodes to read successfully. Then stuck for a long time because of SocketTimeoutException. Log as follows 2020-06-11 16:40:07,832 | INFO | main | Waited 500ms to read from DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; spawning hedged read | DFSInputStream.java:1188 2020-06-11 16:40:08,562 | INFO | main | Waited 500ms to read from DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; spawning hedged read | DFSInputStream.java:1188 2020-06-11 16:40:09,102 | INFO | main | Waited 500ms to read from DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; spawning hedged read | DFSInputStream.java:1188 2020-06-11 16:40:09,642 | INFO | main | Waited 500ms to read from DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; spawning hedged read | DFSInputStream.java:1188 2020-06-11 16:40:10,182 | INFO | main | Waited 500ms to read from DatanodeInfoWithStorage[xx.xx.xx.28:25009,DS-9c843ac6-4ea1-4791-a1af-54c1ae3d5daf,DISK]; spawning hedged read | DFSInputStream.java:1188 2020-06-11 16:40:10,182 | INFO | main | Execution rejected, Executing in current thread | DFSClient.java:3049 2020-06-11 16:40:10,219 | INFO | main | Execution rejected, Executing in current thread | DFSClient.java:3049 2020-06-11 16:50:07,638 | WARN | hedgedRead-0 | I/O error constructing remote block reader. | BlockReaderFactory.java:764 java.net.SocketTimeoutException: 60 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xx.xx.xx.113:62750 remote=/xx.xx.xx.28:25009] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551) at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:418) at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:853) at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:749) at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:379) at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:661) at org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1063) at org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1035) at org.apache.hadoop.hdfs.DFSInputStream$2.call(DFSInputStream.java:1031) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-06-11 16:50:07,638 | WARN | hedgedRead-0 | Connection failure: Failed to connect to /xx.xx.xx.28:25009 for file /testhdfs/test2.jar for block BP-1820384660-xx.xx.xx.74-1585533043013:blk_1082582662_8861386:java.net.SocketTimeoutException: 60 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xx.xx.xx.113:62750 remote=/xx.xx.xx.28:25009] | DFSInputStream.java:1118 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15365) [RBF] findMatching method return wrong result
[ https://issues.apache.org/jira/browse/HDFS-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112692#comment-17112692 ] liuyanyu commented on HDFS-15365: - [~ayushtkn] Sorry, I did not check the open source code, will close this jira. > [RBF] findMatching method return wrong result > - > > Key: HDFS-15365 > URL: https://issues.apache.org/jira/browse/HDFS-15365 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: image-2020-05-20-11-42-12-763.png, > image-2020-05-20-11-55-34-115.png > > > A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, > as follows: > !image-2020-05-20-11-42-12-763.png! > when I used > org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching > to find path hdfs://hacluster/yz0516/abc，the result of > FederationUtil.findMatching(mountTableEntries.iterator(), > /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is > wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15365) [RBF] findMatching method return wrong result
[ https://issues.apache.org/jira/browse/HDFS-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111728#comment-17111728 ] liuyanyu edited comment on HDFS-15365 at 5/20/20, 3:58 AM: --- As I check the code, the part marked in the figure should use actualDest.replaceFirst instead of actualDest.replace [~ayushtkn] pls help to review this, thanks~ !image-2020-05-20-11-55-34-115.png! was (Author: rain_lyy): As I check the code, the part marked in the figure should use actualDest.replaceFirst instead of actualDest.replace [~ayushsaxena] pls help to review this, thanks~ !image-2020-05-20-11-55-34-115.png! > [RBF] findMatching method return wrong result > - > > Key: HDFS-15365 > URL: https://issues.apache.org/jira/browse/HDFS-15365 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: image-2020-05-20-11-42-12-763.png, > image-2020-05-20-11-55-34-115.png > > > A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, > as follows: > !image-2020-05-20-11-42-12-763.png! > when I used > org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching > to find path hdfs://hacluster/yz0516/abc，the result of > FederationUtil.findMatching(mountTableEntries.iterator(), > /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is > wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15365) [RBF] findMatching method return wrong result
[ https://issues.apache.org/jira/browse/HDFS-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111728#comment-17111728 ] liuyanyu edited comment on HDFS-15365 at 5/20/20, 3:56 AM: --- As I check the code, the part marked in the figure should use actualDest.replaceFirst instead of actualDest.replace [~ayushsaxena] pls help to review this, thanks~ !image-2020-05-20-11-55-34-115.png! was (Author: rain_lyy): As I check the code, the part marked in the figure should use actualDest.replaceFirst instead of actualDest.replace [~ayushsaxena] pls help to review this !image-2020-05-20-11-55-34-115.png! > [RBF] findMatching method return wrong result > - > > Key: HDFS-15365 > URL: https://issues.apache.org/jira/browse/HDFS-15365 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: image-2020-05-20-11-42-12-763.png, > image-2020-05-20-11-55-34-115.png > > > A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, > as follows: > !image-2020-05-20-11-42-12-763.png! > when I used > org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching > to find path hdfs://hacluster/yz0516/abc，the result of > FederationUtil.findMatching(mountTableEntries.iterator(), > /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is > wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15365) [RBF] findMatching method return wrong result
[ https://issues.apache.org/jira/browse/HDFS-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111728#comment-17111728 ] liuyanyu commented on HDFS-15365: - As I check the code, the part marked in the figure should use actualDest.replaceFirst instead of actualDest.replace [~ayushsaxena] pls help to review this !image-2020-05-20-11-55-34-115.png! > [RBF] findMatching method return wrong result > - > > Key: HDFS-15365 > URL: https://issues.apache.org/jira/browse/HDFS-15365 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: image-2020-05-20-11-42-12-763.png, > image-2020-05-20-11-48-20-257.png, image-2020-05-20-11-55-34-115.png > > > A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, > as follows: > !image-2020-05-20-11-42-12-763.png! > when I used > org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching > to find path hdfs://hacluster/yz0516/abc，the result of > FederationUtil.findMatching(mountTableEntries.iterator(), > /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is > wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15365) [RBF] findMatching method return wrong result
[ https://issues.apache.org/jira/browse/HDFS-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15365: Attachment: image-2020-05-20-11-55-34-115.png > [RBF] findMatching method return wrong result > - > > Key: HDFS-15365 > URL: https://issues.apache.org/jira/browse/HDFS-15365 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: image-2020-05-20-11-42-12-763.png, > image-2020-05-20-11-55-34-115.png > > > A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, > as follows: > !image-2020-05-20-11-42-12-763.png! > when I used > org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching > to find path hdfs://hacluster/yz0516/abc，the result of > FederationUtil.findMatching(mountTableEntries.iterator(), > /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is > wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15365) [RBF] findMatching method return wrong result
[ https://issues.apache.org/jira/browse/HDFS-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15365: Attachment: (was: image-2020-05-20-11-48-20-257.png) > [RBF] findMatching method return wrong result > - > > Key: HDFS-15365 > URL: https://issues.apache.org/jira/browse/HDFS-15365 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: image-2020-05-20-11-42-12-763.png, > image-2020-05-20-11-55-34-115.png > > > A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, > as follows: > !image-2020-05-20-11-42-12-763.png! > when I used > org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching > to find path hdfs://hacluster/yz0516/abc，the result of > FederationUtil.findMatching(mountTableEntries.iterator(), > /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is > wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15365) [RBF] findMatching method return wrong result
[ https://issues.apache.org/jira/browse/HDFS-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15365: Attachment: image-2020-05-20-11-48-20-257.png > [RBF] findMatching method return wrong result > - > > Key: HDFS-15365 > URL: https://issues.apache.org/jira/browse/HDFS-15365 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: image-2020-05-20-11-42-12-763.png, > image-2020-05-20-11-48-20-257.png > > > A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, > as follows: > !image-2020-05-20-11-42-12-763.png! > when I used > org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching > to find path hdfs://hacluster/yz0516/abc，the result of > FederationUtil.findMatching(mountTableEntries.iterator(), > /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is > wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15365) [RBF] findMatching method return wrong result
liuyanyu created HDFS-15365: --- Summary: [RBF] findMatching method return wrong result Key: HDFS-15365 URL: https://issues.apache.org/jira/browse/HDFS-15365 Project: Hadoop HDFS Issue Type: Bug Components: rbf Affects Versions: 3.1.1 Reporter: liuyanyu Assignee: liuyanyu Attachments: image-2020-05-20-11-42-12-763.png A mount table /hacluster_root -> hdfs://haclsuter/ is setted on the cluster, as follows: !image-2020-05-20-11-42-12-763.png! when I used org.apache.hadoop.hdfs.server.federation.router.FederationUtil#findMatching to find path hdfs://hacluster/yz0516/abc，the result of FederationUtil.findMatching(mountTableEntries.iterator(), /yz0516/abc, hacluster) is /hacluster_root/yz0516/hacluster_root/abc. This is wrong. The correct result should be /hacluster_root/yz0516/abc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104004#comment-17104004 ] liuyanyu commented on HDFS-15243: - Thanks [~ayushtkn] for reviewing, has updated the patch according to your suggestions > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, HDFS-15243.002.patch, > HDFS-15243.003.patch, HDFS-15243.004.patch, HDFS-15243.005.patch, > HDFS-15243.006.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: HDFS-15243.006.patch > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, HDFS-15243.002.patch, > HDFS-15243.003.patch, HDFS-15243.004.patch, HDFS-15243.005.patch, > HDFS-15243.006.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: HDFS-15243.005.patch > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, HDFS-15243.002.patch, > HDFS-15243.003.patch, HDFS-15243.004.patch, HDFS-15243.005.patch, > image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083976#comment-17083976 ] liuyanyu commented on HDFS-15243: - thanks [~ayushtkn] for reviewing, I ignored the compatibility issue. But fs.protected.directories and fs.protected.subdirectories.enable are configrations for the same feature, I think those two configrations should be put together, on the same file. Maybe should put them into Common rather than Hdfs. > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, HDFS-15243.002.patch, > HDFS-15243.003.patch, HDFS-15243.004.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: HDFS-15243.004.patch > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, HDFS-15243.002.patch, > HDFS-15243.003.patch, HDFS-15243.004.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: HDFS-15243.003.patch > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, HDFS-15243.002.patch, > HDFS-15243.003.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: HDFS-15243.002.patch > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, HDFS-15243.002.patch, > image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081955#comment-17081955 ] liuyanyu commented on HDFS-15243: - [~ayushtkn] Thanks for reviewing, will upload the patch again > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: HDFS-15243.001.patch Status: Patch Available (was: Open) > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069355#comment-17069355 ] liuyanyu commented on HDFS-15243: - thanks [~ayushtkn], will consider rename with this logic too, pls assign the Jira to me > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > Attachments: image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15241) Distcp print wrong log info when use -log
[ https://issues.apache.org/jira/browse/HDFS-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15241: Attachment: HDFS-15241.001.patch Status: Patch Available (was: Open) > Distcp print wrong log info when use -log > - > > Key: HDFS-15241 > URL: https://issues.apache.org/jira/browse/HDFS-15241 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: HDFS-15241.001.patch, image-2020-03-25-17-28-33-394.png > > > when run distcp with -log /logpath -v, distcp will print copy status and file > info to /logpath, but print log with wrong file zise. The logs print as > follows: > FILE_COPIED: source=hdfs://ns1/test/stax2-api-3.1.4.jar, size=161867 --> > target=hdfs://ns1/tmp/target/stax2-api-3.1.4.jar, size=0 > As I analysis ,the root cause is as follows: > targrtFileStatus got before copying. So targrtFileStatus is null. Here should > get targrtFileStatus again after file copying. > !image-2020-03-25-17-28-33-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15241) Distcp print wrong log info when use -log
[ https://issues.apache.org/jira/browse/HDFS-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069170#comment-17069170 ] liuyanyu commented on HDFS-15241: - Thanks [~brahmareddy]， will upload patch. > Distcp print wrong log info when use -log > - > > Key: HDFS-15241 > URL: https://issues.apache.org/jira/browse/HDFS-15241 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: image-2020-03-25-17-28-33-394.png > > > when run distcp with -log /logpath -v, distcp will print copy status and file > info to /logpath, but print log with wrong file zise. The logs print as > follows: > FILE_COPIED: source=hdfs://ns1/test/stax2-api-3.1.4.jar, size=161867 --> > target=hdfs://ns1/tmp/target/stax2-api-3.1.4.jar, size=0 > As I analysis ,the root cause is as follows: > targrtFileStatus got before copying. So targrtFileStatus is null. Here should > get targrtFileStatus again after file copying. > !image-2020-03-25-17-28-33-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069165#comment-17069165 ] liuyanyu commented on HDFS-15243: - Yes, we tend to allow read/write but just prevent delete sub-directories under the protected directories. If performance is ok, we even want to protect files from being deleted. My idea is to check if parent directory is in the protected directories list when we try to directory , like this. But this seems to cause performance degradation. waiting for your advice [~ayushtkn] !image-2020-03-28-09-23-31-335.png! > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > Attachments: image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15243: Attachment: image-2020-03-28-09-23-31-335.png > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > Attachments: image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068387#comment-17068387 ] liuyanyu commented on HDFS-15243: - Thanks [~ayushtkn] for reviewing this. In our production cluster, When we protect a directory we mainly to protect the data under this directory not just protect this directory . Because the import business data directories are hundreds or thousands in our cluster, we can not set so many directories to fs.protected.directories. So we want to mark a parent directory of several business directories to protected directory, then the child directories can not be delete or renamed. > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Major > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
liuyanyu created HDFS-15243: --- Summary: Child directory should not be deleted or renamed if parent directory is a protected directory Key: HDFS-15243 URL: https://issues.apache.org/jira/browse/HDFS-15243 Project: Hadoop HDFS Issue Type: Bug Components: 3.1.1 Affects Versions: 3.1.1 Reporter: liuyanyu HDFS-8983 add fs.protected.directories to support protected directories on NameNode. But as I test, when set a parent directory（eg /testA) to protected directory, the child directory (eg /testA/testB) still can be deleted or renamed. When we protect a directory mainly for protecting the data under this directory , So I think the child directory should not be delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15241) Distcp print wrong log info when use -log
[ https://issues.apache.org/jira/browse/HDFS-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066559#comment-17066559 ] liuyanyu commented on HDFS-15241: - [~brahma] Could you pls review this? > Distcp print wrong log info when use -log > - > > Key: HDFS-15241 > URL: https://issues.apache.org/jira/browse/HDFS-15241 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: image-2020-03-25-17-28-33-394.png > > > when run distcp with -log /logpath -v, distcp will print copy status and file > info to /logpath, but print log with wrong file zise. The logs print as > follows: > FILE_COPIED: source=hdfs://ns1/test/stax2-api-3.1.4.jar, size=161867 --> > target=hdfs://ns1/tmp/target/stax2-api-3.1.4.jar, size=0 > As I analysis ,the root cause is as follows: > targrtFileStatus got before copying. So targrtFileStatus is null. Here should > get targrtFileStatus again after file copying. > !image-2020-03-25-17-28-33-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15241) Distcp print wrong log info when use -log
liuyanyu created HDFS-15241: --- Summary: Distcp print wrong log info when use -log Key: HDFS-15241 URL: https://issues.apache.org/jira/browse/HDFS-15241 Project: Hadoop HDFS Issue Type: Improvement Components: distcp Affects Versions: 3.1.1 Reporter: liuyanyu Attachments: image-2020-03-25-17-28-33-394.png when run distcp with -log /logpath -v, distcp will print copy status and file info to /logpath, but print log with wrong file zise. The logs print as follows: FILE_COPIED: source=hdfs://ns1/test/stax2-api-3.1.4.jar, size=161867 --> target=hdfs://ns1/tmp/target/stax2-api-3.1.4.jar, size=0 As I analysis ,the root cause is as follows: targrtFileStatus got before copying. So targrtFileStatus is null. Here should get targrtFileStatus again after file copying. !image-2020-03-25-17-28-33-394.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15194) ERROR log print wrong user info when run distcp
[ https://issues.apache.org/jira/browse/HDFS-15194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046028#comment-17046028 ] liuyanyu commented on HDFS-15194: - [~brahmareddy] thanks for reviewing. This is indeed a duplicate of HDFS-13626. I didn't find HDFS-13626 before. > ERROR log print wrong user info when run distcp > --- > > Key: HDFS-15194 > URL: https://issues.apache.org/jira/browse/HDFS-15194 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: distcp.log, image-2020-02-26-14-10-19-654.png > > > Use user test which belong to group hadoop to run distcp with -pbugpaxt , cp > a directory which owner is super, distcp runs failed error log as follows: > 2020-02-26 11:17:02,755 INFO [IPC Server handler 5 on 27101] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from > attempt_1582635453769_0003_m_21_0: Error: > org.apache.hadoop.security.AccessControlException: User super is not a super > user (non-super user cannot change owner). > at > org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:85) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1927) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:870) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:566) > at > ... > Currnet user is test, not super, the log print wrong user info > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15194) ERROR log print wrong user info when run distcp
[ https://issues.apache.org/jira/browse/HDFS-15194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045178#comment-17045178 ] liuyanyu commented on HDFS-15194: - As I analysis ,the root cause is as follows: username should change to pc.getUser() !image-2020-02-26-14-10-19-654.png! > ERROR log print wrong user info when run distcp > --- > > Key: HDFS-15194 > URL: https://issues.apache.org/jira/browse/HDFS-15194 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: distcp.log, image-2020-02-26-14-10-19-654.png > > > Use user test which belong to group hadoop to run distcp with -pbugpaxt , cp > a directory which owner is super, distcp runs failed error log as follows: > 2020-02-26 11:17:02,755 INFO [IPC Server handler 5 on 27101] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from > attempt_1582635453769_0003_m_21_0: Error: > org.apache.hadoop.security.AccessControlException: User super is not a super > user (non-super user cannot change owner). > at > org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:85) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1927) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:870) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:566) > at > ... > Currnet user is test, not super, the log print wrong user info > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15194) ERROR log print wrong user info when run distcp
[ https://issues.apache.org/jira/browse/HDFS-15194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15194: Attachment: image-2020-02-26-14-10-19-654.png > ERROR log print wrong user info when run distcp > --- > > Key: HDFS-15194 > URL: https://issues.apache.org/jira/browse/HDFS-15194 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: distcp.log, image-2020-02-26-14-10-19-654.png > > > Use user test which belong to group hadoop to run distcp with -pbugpaxt , cp > a directory which owner is super, distcp runs failed error log as follows: > 2020-02-26 11:17:02,755 INFO [IPC Server handler 5 on 27101] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from > attempt_1582635453769_0003_m_21_0: Error: > org.apache.hadoop.security.AccessControlException: User super is not a super > user (non-super user cannot change owner). > at > org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:85) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1927) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:870) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:566) > at > ... > Currnet user is test, not super, the log print wrong user info > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15194) ERROR log print wrong user info when run distcp
[ https://issues.apache.org/jira/browse/HDFS-15194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuyanyu updated HDFS-15194: Attachment: distcp.log > ERROR log print wrong user info when run distcp > --- > > Key: HDFS-15194 > URL: https://issues.apache.org/jira/browse/HDFS-15194 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.1.1 >Reporter: liuyanyu >Priority: Minor > Attachments: distcp.log > > > Use user test which belong to group hadoop to run distcp with -pbugpaxt , cp > a directory which owner is super, distcp runs failed error log as follows: > 2020-02-26 11:17:02,755 INFO [IPC Server handler 5 on 27101] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from > attempt_1582635453769_0003_m_21_0: Error: > org.apache.hadoop.security.AccessControlException: User super is not a super > user (non-super user cannot change owner). > at > org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:85) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1927) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:870) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:566) > at > ... > Currnet user is test, not super, the log print wrong user info > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15194) ERROR log print wrong user info when run distcp
liuyanyu created HDFS-15194: --- Summary: ERROR log print wrong user info when run distcp Key: HDFS-15194 URL: https://issues.apache.org/jira/browse/HDFS-15194 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.1.1 Reporter: liuyanyu Use user test which belong to group hadoop to run distcp with -pbugpaxt , cp a directory which owner is super, distcp runs failed error log as follows: 2020-02-26 11:17:02,755 INFO [IPC Server handler 5 on 27101] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1582635453769_0003_m_21_0: Error: org.apache.hadoop.security.AccessControlException: User super is not a super user (non-super user cannot change owner). at org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setOwner(FSDirAttrOp.java:85) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setOwner(FSNamesystem.java:1927) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setOwner(NameNodeRpcServer.java:870) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setOwner(ClientNamenodeProtocolServerSideTranslatorPB.java:566) at ... Currnet user is test, not super, the log print wrong user info -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
