[jira] [Updated] (HDFS-17151) EC: Fix wrong metadata in BlockInfoStriped after recovery
[ https://issues.apache.org/jira/browse/HDFS-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17151: -- Component/s: erasure-coding Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > EC: Fix wrong metadata in BlockInfoStriped after recovery > - > > Key: HDFS-17151 > URL: https://issues.apache.org/jira/browse/HDFS-17151 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When the datanode completes a block recovery, it will call > `commitBlockSynchronization` method to notify NN the new locations of the > block. For a EC block group, NN determines the index of each internal block > based on the position of the DatanodeID in the parameter `newtargets`. > If the internal blocks written by the client don't have continuous indices, > the current datanode code might cause NN to record incorrect block metadata. > For simplicity, let's take RS (3,2) as an example. The timeline of the > problem is as follows: > 1. The client plans to write internal blocks with indices [0,1,2,3,4] to > datanode [dn0, dn1, dn2, dn3, dn4] respectively. But dn1 is unable to > connect, so the client only writes data to the remaining 4 datanodes; > 2. Client crashes; > 3. NN fails over; > 4. Now the content of `uc. getExpectedStorageLocations()` completely depends > on block reports, and now it is ; > 5. When the lease expires hard limit, NN issues a block recovery command; > 6. Datanode that receives the recovery command fills `DatanodeID [] newLocs` > with [dn0, null, dn2, dn3, dn4]; > 7. The serialization process filters out null values, so the parameters > passed to NN become [dn0, dn2, dn3, dn4]; > 8. NN mistakenly believes that dn2 stores an internal block with index 1, dn3 > stores an internal block with index 2, and so on. > The above timeline is just an example, and there are other situations that > may result in the same error, such as an update pipeline occurs on the client > side. We should fix this bug. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17150) EC: Fix the bug of failed lease recovery.
[ https://issues.apache.org/jira/browse/HDFS-17150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17150: -- Component/s: erasure-coding Target Version/s: 3.4.0, 3.3.9 Affects Version/s: 3.4.0 3.3.9 > EC: Fix the bug of failed lease recovery. > - > > Key: HDFS-17150 > URL: https://issues.apache.org/jira/browse/HDFS-17150 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.4.0, 3.3.9 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > If the client crashes without writing the minimum number of internal blocks > required by the EC policy, the lease recovery process for the corresponding > unclosed file may continue to fail. Taking RS(6,3) policy as an example, the > timeline is as follows: > 1. The client writes some data to only 5 datanodes; > 2. Client crashes; > 3. NN fails over; > 4. Now the result of `uc.getNumExpectedLocations()` completely depends on > block report, and there are 5 datanodes reporting internal blocks; > 5. When the lease expires hard limit, NN issues a block recovery command; > 6. The datanode checks the command and finds that the number of internal > blocks is insufficient, resulting in an error and recovery failure; > 7. The lease expires hard limit again, and NN issues a block recovery command > again, but the recovery fails again.. > When the number of internal blocks written by the client is less than 6, the > block group is actually unrecoverable. We should equate this situation to the > case where the number of replicas is 0 when processing replica files, i.e., > directly remove the last block group and close the file. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17140) Revisit the BPOfferService.reportBadBlocks() method.
[ https://issues.apache.org/jira/browse/HDFS-17140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17140: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Revisit the BPOfferService.reportBadBlocks() method. > > > Key: HDFS-17140 > URL: https://issues.apache.org/jira/browse/HDFS-17140 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: Liangjun He >Assignee: Liangjun He >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > The current BPOfferService.reportBadBlocks() method can be optimized by > moving the creation of the rbbAction object outside the loop. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17148) RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL
[ https://issues.apache.org/jira/browse/HDFS-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17148: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: SQLDelegationTokenSecretManager must cleanup expired tokens in SQL > --- > > Key: HDFS-17148 > URL: https://issues.apache.org/jira/browse/HDFS-17148 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The SQLDelegationTokenSecretManager fetches tokens from SQL and stores them > temporarily in a memory cache with a short TTL. The ExpiredTokenRemover in > AbstractDelegationTokenSecretManager runs periodically to cleanup any expired > tokens from the cache, but most tokens have been evicted automatically per > the TTL configuration. This leads to many expired tokens in the SQL database > that should be cleaned up. > The SQLDelegationTokenSecretManager should find expired tokens in SQL instead > of in the memory cache when running the periodic cleanup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17145) Fix description of property dfs.namenode.file.close.num-committed-allowed.
[ https://issues.apache.org/jira/browse/HDFS-17145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17145: -- Hadoop Flags: Reviewed > Fix description of property dfs.namenode.file.close.num-committed-allowed. > -- > > Key: HDFS-17145 > URL: https://issues.apache.org/jira/browse/HDFS-17145 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > > Normally a file can only be closed with all its blocks are complete. > But in hdfs-default.xml, it it committed. We should fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17144) Remove incorrect comment in method storeAllocatedBlock
[ https://issues.apache.org/jira/browse/HDFS-17144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17144: -- Hadoop Flags: Reviewed > Remove incorrect comment in method storeAllocatedBlock > -- > > Key: HDFS-17144 > URL: https://issues.apache.org/jira/browse/HDFS-17144 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > > As title described, remove incorrect comment in method storeAllocatedBlock. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17139) RBF: For the doc of the class RouterAdminProtocolTranslatorPB, it describes the function of the class ClientNamenodeProtocolTranslatorPB
[ https://issues.apache.org/jira/browse/HDFS-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17139: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: For the doc of the class RouterAdminProtocolTranslatorPB, it describes > the function of the class ClientNamenodeProtocolTranslatorPB > > > Key: HDFS-17139 > URL: https://issues.apache.org/jira/browse/HDFS-17139 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HDFS-17139.001.patch, HDFS-17139.patch > > > For the doc of the class RouterAdminProtocolTranslatorPB, it describes the > function of the class ClientNamenodeProtocolTranslatorPB. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17137) Standby/Observer NameNode skip to handle redundant replica block logic when set decrease replication.
[ https://issues.apache.org/jira/browse/HDFS-17137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17137: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Standby/Observer NameNode skip to handle redundant replica block logic when > set decrease replication. > -- > > Key: HDFS-17137 > URL: https://issues.apache.org/jira/browse/HDFS-17137 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Standby/Observer NameNode should not handle redundant replica block logic > when set decrease replication. > At present, when call setReplication to execute the logic of decrease > replication, > * ActiveNameNode will call the BlockManager#processExtraRedundancyBlock > method to select the dn of the redundant replica , will add to the > excessRedundancyMap and add to invalidateBlocks (RedundancyMonitor will be > scheduled to delete the block on dn). > * Then the StandyNameNode or ObserverNameNode load editlog and apply the > SetReplicationOp, if the dn of the replica to be deleted has not yet > performed incremental block report, > here also will BlockManager#processExtraRedundancyBlock method be called here > to select the dn of the redundant replica and add it to the > excessRedundancyMap (here selected the redundant dn may be inconsistent with > the dn selected in the active namenode). > In excessRedundancyMap exist dn maybe affects the dn decommission, resulting > can not to complete decommission dn operation in Standy/ObserverNameNode. > The specific cases are as follows: > For example a file is 3 replica (d1,d2,d3) and call setReplication set file > to 2 replica. > * ActiveNameNode select d1 with redundant replicas to add > toexcessRedundancyMap and invalidateBlocks. > * StandyNameNode replays SetReplicationOp (at this time, d1 has not yet > executed incremental block report), so here maybe selected redundant replica > dn are inconsistent with ActiveNameNode, such as select d2 to add > excessRedundancyMap. > * At this time, d1 completes deleting the block for incremental block report. > * The DN list for this block in ActiveNameNode includes d2 and d3 (delete d1 > from in the excessRedundancyMap when processing the incremental block report > ). > * The DN list for this block in StandyNameNode includes d2 and d3 (can not > delete d2 from in the excessRedundancyMap when processing the incremental > block report). > At this time, execute the decommission operation on d3. > * ActiveNameNode will select a new node d4 to copy the replica, and d4 will > run incrementally block report. > * The DN list for this block in ActiveNameNode includes d2 and > d3(decommissioning status),d4, then d3 can to decommissioned normally. > * The DN list for this block in StandyNameNode is d3 (decommissioning > status), d2 (redundant status), d4. > since the requirements for two live replica are not met, d3 cannot be > decommissioned at this time. > Therefore, StandyNameNode or ObserverNameNode considers not process redundant > replicas logic when call setReplication. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working
[ https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17138: -- Component/s: rbf Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: We changed the hadoop.security.auth_to_local configuration of one > router, the other routers stopped working > > > Key: HDFS-17138 > URL: https://issues.apache.org/jira/browse/HDFS-17138 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 > Environment: hadoop 3.3.0 >Reporter: Xiping Zhang >Assignee: Xiping Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-08-02-16-20-34-454.png, > image-2023-08-03-10-32-03-457.png > > > other routers error log： > !image-2023-08-02-16-20-34-454.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17136) Fix annotation description and typo in BlockPlacementPolicyDefault Class
[ https://issues.apache.org/jira/browse/HDFS-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17136: -- Affects Version/s: 3.4.0 > Fix annotation description and typo in BlockPlacementPolicyDefault Class > > > Key: HDFS-17136 > URL: https://issues.apache.org/jira/browse/HDFS-17136 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: huangzhaobo >Assignee: huangzhaobo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix annotation description and typo in BlockPlacementPolicyDefault Class -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17136) Fix annotation description and typo in BlockPlacementPolicyDefault Class
[ https://issues.apache.org/jira/browse/HDFS-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17136: -- Hadoop Flags: Reviewed > Fix annotation description and typo in BlockPlacementPolicyDefault Class > > > Key: HDFS-17136 > URL: https://issues.apache.org/jira/browse/HDFS-17136 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.4.0 >Reporter: huangzhaobo >Assignee: huangzhaobo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix annotation description and typo in BlockPlacementPolicyDefault Class -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17136) Fix annotation description and typo in BlockPlacementPolicyDefault Class
[ https://issues.apache.org/jira/browse/HDFS-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17136: -- Component/s: block placement > Fix annotation description and typo in BlockPlacementPolicyDefault Class > > > Key: HDFS-17136 > URL: https://issues.apache.org/jira/browse/HDFS-17136 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.4.0 >Reporter: huangzhaobo >Assignee: huangzhaobo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix annotation description and typo in BlockPlacementPolicyDefault Class -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17136) Fix annotation description and typo in BlockPlacementPolicyDefault Class
[ https://issues.apache.org/jira/browse/HDFS-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17136: -- Target Version/s: 3.4.0 > Fix annotation description and typo in BlockPlacementPolicyDefault Class > > > Key: HDFS-17136 > URL: https://issues.apache.org/jira/browse/HDFS-17136 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: huangzhaobo >Assignee: huangzhaobo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix annotation description and typo in BlockPlacementPolicyDefault Class -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17135) Update fsck -blockId to display excess state info of blocks
[ https://issues.apache.org/jira/browse/HDFS-17135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17135: -- Component/s: namnode > Update fsck -blockId to display excess state info of blocks > > > Key: HDFS-17135 > URL: https://issues.apache.org/jira/browse/HDFS-17135 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namnode >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Update fsck -blockId to display excess state info of blocks -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17133) TestFsDatasetImpl missing null check when cleaning up
[
https://issues.apache.org/jira/browse/HDFS-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17133:
--
Component/s: test
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> TestFsDatasetImpl missing null check when cleaning up
> -
>
> Key: HDFS-17133
> URL: https://issues.apache.org/jira/browse/HDFS-17133
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0
>Reporter: ConfX
>Assignee: ConfX
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: reproduce.sh
>
>
> h2. What happened
> I turned on {{dfs.namenode.quota.init-threads=1468568631}} and then the test
> {{{}org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl#testMoveBlockFailure{}}}fails
> with null pointer.
> h2. Where's the problem
> In the clean up part of the test:
> {noformat}
> } finally {
> if (cluster.isClusterUp()) {
> cluster.shutdown();
> }
> }{noformat}
> if cluster is null, the test would directly fail with a null pointer
> exception and hiding potentially the actual failure.
> h2. How to reproduce
> # set {{{}dfs.namenode.quota.init-threads{}}}={{{}1468568631 {}}}
> # run
> {{org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl#testMoveBlockFailure}}
> you should observe
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testMoveBlockFailure(TestFsDatasetImpl.java:1005){noformat}
> For an easy reproduction, run the reproduce.sh in the attachment.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17135) Update fsck -blockId to display excess state info of blocks
[ https://issues.apache.org/jira/browse/HDFS-17135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17135: -- Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Update fsck -blockId to display excess state info of blocks > > > Key: HDFS-17135 > URL: https://issues.apache.org/jira/browse/HDFS-17135 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namnode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Update fsck -blockId to display excess state info of blocks -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17134) RBF: Fix duplicate results of getListing through Router.
[ https://issues.apache.org/jira/browse/HDFS-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17134: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: Fix duplicate results of getListing through Router. > > > Key: HDFS-17134 > URL: https://issues.apache.org/jira/browse/HDFS-17134 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The result of `getListing` in NameNode are sorted based on `byte[]`, while > the Router side is based on `String`. If there are special characters in > path, the sorting result of the router will be inconsistent with the > namenode. This may result in duplicate `getListing` results obtained by the > client due to wrong `startAfter` parameter. > For exemple, namenode returns [path1, path2, path3] for a `getListing` > request, while router returns [path1, path3, path2] to client. Then client > will pass `path2` as `startAfter` at the next iteration, so it will receive > `path3` again. > We need to fix the Router code so that the order of its result is the same as > NameNode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17122) Rectify the table length discrepancy in the DataNode UI.
[
https://issues.apache.org/jira/browse/HDFS-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17122:
--
Component/s: ui
Target Version/s: 3.4.0
> Rectify the table length discrepancy in the DataNode UI.
>
>
> Key: HDFS-17122
> URL: https://issues.apache.org/jira/browse/HDFS-17122
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ui
>Affects Versions: 3.4.0
>Reporter: Hualong Zhang
>Assignee: Hualong Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: image-2023-07-25-18-12-10-582.png
>
>
> The hidden column settings in *table-datanodes.dataTable* have caused an
> error in the calculation of table length in {*}dataTable{*}.
> !image-2023-07-25-18-12-10-582.png|width=798,height=318!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17119) RBF: Logger fix for StateStoreMySQLImpl
[ https://issues.apache.org/jira/browse/HDFS-17119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17119: -- Component/s: rbf Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: Logger fix for StateStoreMySQLImpl > --- > > Key: HDFS-17119 > URL: https://issues.apache.org/jira/browse/HDFS-17119 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17117) Print reconstructionQueuesInitProgress periodically when BlockManager processMisReplicatesAsync.
[ https://issues.apache.org/jira/browse/HDFS-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17117: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Print reconstructionQueuesInitProgress periodically when BlockManager > processMisReplicatesAsync. > > > Key: HDFS-17117 > URL: https://issues.apache.org/jira/browse/HDFS-17117 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > BlockManager#processMisReplicatesAsync can periodically print > reconstructionQueuesInitProgress, so that the admin can get the progress of > the replication queues initialisation. > Ready to add logs and metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17116) RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService.
[
https://issues.apache.org/jira/browse/HDFS-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17116:
--
Component/s: rbf
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> RBF: Update invoke millisecond time as monotonicNow() in
> RouterSafemodeService.
> ---
>
> Key: HDFS-17116
> URL: https://issues.apache.org/jira/browse/HDFS-17116
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The following exceptions occurred in our online environment:
> # After the machine restarts, the system time is abnormal, is a time in the
> future
> # After starting the router, there is log "safemode exit for 24981702
> milliseconds...", which has been in the safemode state,
> this is mainly because the startupTime is recorded as the future system time
> when router is started at this time, and the system time returns to normal
> soon, resulting in a negative delta,
> at this time, the service can only be restored by restart the router service.
> The relevant logs are:
> {code:java}
> 2023-07-15 03:15:49,276 INFO ipc.Server xxx
> 2023-07-15 11:21:03,785 INFO router.DFSRouter (LogAdapter.java:info(51))
> [main] - STARTUP_MSG:
> /
> STARTUP_MSG: Starting Router
> ...
> 2023-07-15 11:21:51,325 INFO xxx
> 2023-07-15 03:22:00,257 INFO xxx
> 2023-07-15 03:22:29,829 INFO router.RouterSafemodeService
> (RouterSafemodeService.java:periodicInvoke(167)) [RouterSafemodeService-0] -
> Delaying safemode exit for 28761777 milliseconds...
> {code}
> Maybe we can be compatible with this case at the code level, and reset the
> startupTime and enterSafeModeTime in the case of a negative delta,
> which can ensure that the router service can also exit the safemode state
> normally after the system time returns to normal.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17111) RBF: Optimize msync to only call nameservices that have observer reads enabled.
[ https://issues.apache.org/jira/browse/HDFS-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17111: -- Component/s: rbf Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: Optimize msync to only call nameservices that have observer reads > enabled. > --- > > Key: HDFS-17111 > URL: https://issues.apache.org/jira/browse/HDFS-17111 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Right now when a client MSYNCs to the router, the call is fanned out to all > nameservices. We only need to proxy the msync to nameservices that have > observer reads configured. > We can do this either by adding a new config for the admin to specify which > nameservices have CRS configured, or we can try to automatically detect these. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17112) Show decommission duration in JMX and HTML
[ https://issues.apache.org/jira/browse/HDFS-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17112: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Show decommission duration in JMX and HTML > -- > > Key: HDFS-17112 > URL: https://issues.apache.org/jira/browse/HDFS-17112 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Expose decommission duration time in JMX page. It's a very useful info when > decommissioning a batch of datanodes in a cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17115) HttpFS Add Support getErasureCodeCodecs API
[
https://issues.apache.org/jira/browse/HDFS-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17115:
--
Target Version/s: 3.4.0
> HttpFS Add Support getErasureCodeCodecs API
> ---
>
> Key: HDFS-17115
> URL: https://issues.apache.org/jira/browse/HDFS-17115
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Affects Versions: 3.4.0
>Reporter: Hualong Zhang
>Assignee: Hualong Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We should ensure that *WebHDFS* remains synchronized with {*}HttpFS{*}, as
> the former has already implemented the *getErasureCodeCodecs* interface.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17093) Fix block report lease issue to avoid missing some storages report.
[
https://issues.apache.org/jira/browse/HDFS-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17093:
--
Target Version/s: 3.4.0
> Fix block report lease issue to avoid missing some storages report.
> ---
>
> Key: HDFS-17093
> URL: https://issues.apache.org/jira/browse/HDFS-17093
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.4
>Reporter: Yanlei Yu
>Assignee: Yanlei Yu
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> In our cluster of 800+ nodes, after restarting the namenode, we found that
> some datanodes did not report enough blocks, causing the namenode to stay in
> secure mode for a long time after restarting because of incomplete block
> reporting
> I found in the logs of the datanode with incomplete block reporting that the
> first FBR attempt failed, possibly due to namenode stress, and then a second
> FBR attempt was made as follows:
> {code:java}
>
> 2023-07-17 11:29:28,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unsuccessfully sent block report 0x6237a52c1e817e, containing 12 storage
> report(s), of which we sent 1. The reports had 1099057 total blocks and used
> 1 RPC(s). This took 294 msec to generate and 101721 msecs for RPC and NN
> processing. Got back no commands.
> 2023-07-17 11:37:04,014 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Successfully sent block report 0x62382416f3f055, containing 12 storage
> report(s), of which we sent 12. The reports had 1099048 total blocks and used
> 12 RPC(s). This took 295 msec to generate and 11647 msecs for RPC and NN
> processing. Got back no commands. {code}
> There's nothing wrong with that. Retry the send if it fails But on the
> namenode side of the logic:
> {code:java}
> if (namesystem.isInStartupSafeMode()
> && !StorageType.PROVIDED.equals(storageInfo.getStorageType())
> && storageInfo.getBlockReportCount() > 0) {
> blockLog.info("BLOCK* processReport 0x{} with lease ID 0x{}: "
> + "discarded non-initial block report from {}"
> + " because namenode still in startup phase",
> strBlockReportId, fullBrLeaseId, nodeID);
> blockReportLeaseManager.removeLease(node);
> return !node.hasStaleStorages();
> } {code}
> When a disk was identified as the report is not the first time, namely
> storageInfo. GetBlockReportCount > 0, Will remove the ticket from the
> datanode, lead to a second report failed because no lease
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17105) mistakenly purge editLogs even after it is empty in NNStorageRetentionManager
[
https://issues.apache.org/jira/browse/HDFS-17105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17105:
--
Component/s: namanode
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> mistakenly purge editLogs even after it is empty in NNStorageRetentionManager
> --
>
> Key: HDFS-17105
> URL: https://issues.apache.org/jira/browse/HDFS-17105
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: ConfX
>Assignee: ConfX
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: reproduce.sh
>
>
> h2. What happened:
> Got {{IndexOutOfBoundsException}} after setting
> {{dfs.namenode.max.extra.edits.segments.retained}} to a negative value and
> purging old record with {{{}NNStorageRetentionManager{}}}.
> h2. Where's the bug:
> In line 156 of {{{}NNStorageRetentionManager{}}}, the manager trims
> {{editLogs}} until it is under the {{{}maxExtraEditsSegmentsToRetain{}}}:
> {noformat}
> while (editLogs.size() > maxExtraEditsSegmentsToRetain) {
> purgeLogsFrom = editLogs.get(0).getLastTxId() + 1;
> editLogs.remove(0);
> }{noformat}
> However, if {{dfs.namenode.max.extra.edits.segments.retained}} is set to
> below 0 the size of {{editLogs}} would never be below, resulting in
> ultimately {{editLog.size()=0}} and thus {{editLogs.get(0)}} is out of range.
> h2. How to reproduce:
> (1) Set {{dfs.namenode.max.extra.edits.segments.retained}} to -1974676133
> (2) Run test:
> {{org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager#testNoLogs}}
> h2. Stacktrace:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
> at
> java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
> at
> java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
> at
> java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
> at java.base/java.util.Objects.checkIndex(Objects.java:372)
> at java.base/java.util.ArrayList.get(ArrayList.java:459)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:157)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager.runTest(TestNNStorageRetentionManager.java:299)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionManager.testNoLogs(TestNNStorageRetentionManager.java:143){noformat}
> For an easy reproduction, run the reproduce.sh in the attachment.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17094) EC: Fix bug in block recovery when there are stale datanodes
[ https://issues.apache.org/jira/browse/HDFS-17094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17094: -- Component/s: erasure-coding Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > EC: Fix bug in block recovery when there are stale datanodes > > > Key: HDFS-17094 > URL: https://issues.apache.org/jira/browse/HDFS-17094 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When a block recovery occurs, `RecoveryTaskStriped` in datanode expects > `rBlock.getLocations()` and `rBlock. getBlockIndices()` to be in one-to-one > correspondence. However, if there are locations in stale state when NameNode > handles heartbeat, this correspondence will be disrupted. In detail, there is > no stale location in `recoveryLocations`, but the block indices array is > still complete (i.e. contains the indices of all the locations). This will > cause `BlockRecoveryWorker.RecoveryTaskStriped#recover` to generate a wrong > internal block ID, and the corresponding datanode cannot find the replica, > thus making the recovery process fail. This bug needs to be fixed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17088) Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md
[ https://issues.apache.org/jira/browse/HDFS-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17088: -- Component/s: dfsadmin Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md > - > > Key: HDFS-17088 > URL: https://issues.apache.org/jira/browse/HDFS-17088 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfsadmin >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Improve the debug verifyEC and dfsrouteradmin commands in HDFSCommands.md > * the debug verifyEC add new params description, such as [-blockId ] > [-skipFailureBlocks]. > * dfsrouteradmin added a new command about -addAll. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17087) Add Throttler for datanode reading block
[
https://issues.apache.org/jira/browse/HDFS-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17087:
--
Component/s: datanode
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Add Throttler for datanode reading block
> -
>
> Key: HDFS-17087
> URL: https://issues.apache.org/jira/browse/HDFS-17087
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.4.0
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> DataXceiver#readBlock code is:
> {code:java}
> ...
> read = blockSender.sendBlock(out, baseStream, null); // send data
> ..
> {code}
> Current DataXceiver#readBlock doesn't throttler, maybe we can support
> throttle for reading block satisfied catering for specific scenarios.
> Default read throttler value is still null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17086) Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota.
[ https://issues.apache.org/jira/browse/HDFS-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17086: -- Component/s: test Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix the parameter settings in TestDiskspaceQuotaUpdate#updateCountForQuota. > --- > > Key: HDFS-17086 > URL: https://issues.apache.org/jira/browse/HDFS-17086 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17082) Add documentation for provisionSnapshotTrash command to HDFSCommands.md and HdfsSnapshots.md
[ https://issues.apache.org/jira/browse/HDFS-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17082: -- Component/s: documentation Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Add documentation for provisionSnapshotTrash command to HDFSCommands.md and > HdfsSnapshots.md > - > > Key: HDFS-17082 > URL: https://issues.apache.org/jira/browse/HDFS-17082 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HDFS-15607 and HDFS-15997 introduced provisionSnapshotTrash should add it to > the document. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17083) Support getErasureCodeCodecs API in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-17083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17083: -- Target Version/s: 3.4.0 > Support getErasureCodeCodecs API in WebHDFS > --- > > Key: HDFS-17083 > URL: https://issues.apache.org/jira/browse/HDFS-17083 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-07-12-22-52-15-954.png > > > WebHDFS should support getErasureCodeCodecs: > !image-2023-07-12-22-52-15-954.png|width=799,height=210! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17081) EC: Add logic for striped blocks in isSufficientlyReplicated
[
https://issues.apache.org/jira/browse/HDFS-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17081:
--
Component/s: erasure-coding
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> EC: Add logic for striped blocks in isSufficientlyReplicated
>
>
> Key: HDFS-17081
> URL: https://issues.apache.org/jira/browse/HDFS-17081
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.4.0
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Append ec file check if a block is replicated to at least the minimum
> replication need consider ec block.
> currently only the minimum replication of the replica is considered, the code
> is as follows:
> {code:java}
> /**
>* Check if a block is replicated to at least the minimum replication.
>*/
> public boolean isSufficientlyReplicated(BlockInfo b) {
> // Compare against the lesser of the minReplication and number of live
> DNs.
> final int liveReplicas = countNodes(b).liveReplicas();
> if (liveReplicas >= minReplication) {
> return true;
> }
> // getNumLiveDataNodes() is very expensive and we minimize its use by
> // comparing with minReplication first.
> return liveReplicas >= getDatanodeManager().getNumLiveDataNodes();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17075) Reconfig disk balancer parameters for datanode
[ https://issues.apache.org/jira/browse/HDFS-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17075: -- Component/s: datanode Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Reconfig disk balancer parameters for datanode > -- > > Key: HDFS-17075 > URL: https://issues.apache.org/jira/browse/HDFS-17075 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Rolling restart datanodes takes long time, can make disk balanacer parameters > dfs.disk.balancer.enabled and dfs.disk.balancer.plan.valid.interval in > datanode reconfigurable to facilitate cluster operation and maintenance. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17076) Remove the unused method isSlownodeByNameserviceId in DataNode
[ https://issues.apache.org/jira/browse/HDFS-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17076: -- Component/s: datanode Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Remove the unused method isSlownodeByNameserviceId in DataNode > -- > > Key: HDFS-17076 > URL: https://issues.apache.org/jira/browse/HDFS-17076 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Remove the unused method isSlownodeByNameserviceId() in DataNode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17070) Remove unused import in DataNodeMetricHelper.java.
[ https://issues.apache.org/jira/browse/HDFS-17070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17070: -- Component/s: datanode > Remove unused import in DataNodeMetricHelper.java. > -- > > Key: HDFS-17070 > URL: https://issues.apache.org/jira/browse/HDFS-17070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > > Remove unused import in DataNodeMetricHelper.java. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17073) Enhance the warning message output for BlockGroupNonStripedChecksumComputer#compute
[ https://issues.apache.org/jira/browse/HDFS-17073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17073: -- Component/s: hdfs Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Enhance the warning message output for > BlockGroupNonStripedChecksumComputer#compute > --- > > Key: HDFS-17073 > URL: https://issues.apache.org/jira/browse/HDFS-17073 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Consider improving the log output of the warning messages generated by > BlockGroupNonStripedChecksumComputer when calling checksumBlock, to make it > easier to locate the block information where an exception occurs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17070) Remove unused import in DataNodeMetricHelper.java.
[ https://issues.apache.org/jira/browse/HDFS-17070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17070: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Remove unused import in DataNodeMetricHelper.java. > -- > > Key: HDFS-17070 > URL: https://issues.apache.org/jira/browse/HDFS-17070 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Trivial > Labels: pull-request-available > Fix For: 3.4.0 > > > Remove unused import in DataNodeMetricHelper.java. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17068) Datanode should record last directory scan time.
[ https://issues.apache.org/jira/browse/HDFS-17068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17068: -- Affects Version/s: 3.4.0 > Datanode should record last directory scan time. > > > Key: HDFS-17068 > URL: https://issues.apache.org/jira/browse/HDFS-17068 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > I think it is useful for us to record last directory scan time for one > datanode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17065) Fix typos in hadoop-hdfs-project
[ https://issues.apache.org/jira/browse/HDFS-17065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17065: -- Component/s: hdfs Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix typos in hadoop-hdfs-project > > > Key: HDFS-17065 > URL: https://issues.apache.org/jira/browse/HDFS-17065 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17068) Datanode should record last directory scan time.
[ https://issues.apache.org/jira/browse/HDFS-17068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17068: -- Component/s: datanode Target Version/s: 3.4.0 > Datanode should record last directory scan time. > > > Key: HDFS-17068 > URL: https://issues.apache.org/jira/browse/HDFS-17068 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > I think it is useful for us to record last directory scan time for one > datanode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17064) Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter
[ https://issues.apache.org/jira/browse/HDFS-17064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17064: -- Component/s: balancer documentation Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Document the usage of the new Balancer "sortTopNodes" and > "hotBlockTimeInterval" parameter > -- > > Key: HDFS-17064 > URL: https://issues.apache.org/jira/browse/HDFS-17064 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer, documentation >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17063) Support to configure different capacity reserved for each disk of DataNode.
[
https://issues.apache.org/jira/browse/HDFS-17063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17063:
--
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
> Support to configure different capacity reserved for each disk of DataNode.
> ---
>
> Key: HDFS-17063
> URL: https://issues.apache.org/jira/browse/HDFS-17063
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, hdfs
>Affects Versions: 3.3.6
>Reporter: Jiale Qi
>Assignee: Jiale Qi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Now _dfs.datanode.du.reserved_ takes effect for all directory of a datanode.
> This issue allows cluster administrator to configure
> {_}dfs.datanode.du.reserved./data/hdfs1/data{_}, which only take effect for a
> specific directory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17056) EC: Fix verifyClusterSetup output in case of an invalid param.
[
https://issues.apache.org/jira/browse/HDFS-17056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17056:
--
Component/s: erasure-coding
(was: ec)
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> EC: Fix verifyClusterSetup output in case of an invalid param.
> --
>
> Key: HDFS-17056
> URL: https://issues.apache.org/jira/browse/HDFS-17056
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.4.0
>Reporter: Ayush Saxena
>Assignee: huangzhaobo
>Priority: Major
> Labels: newbie, pull-request-available
> Fix For: 3.4.0
>
>
> {code:java}
> bin/hdfs ec -verifyClusterSetup XOR-2-1-1024k
> 9 DataNodes are required for the erasure coding policies: RS-6-3-1024k,
> XOR-2-1-1024k. The number of DataNodes is only 3. {code}
> verifyClusterSetup requires -policy then the name of policies, else it
> defaults to all enabled policies.
> In case there are additional invalid options it silently ignores them, unlike
> other EC commands which throws out Too Many Argument exception.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17057) RBF: Add DataNode maintenance states to Federation UI
[ https://issues.apache.org/jira/browse/HDFS-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17057: -- Component/s: rbf Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: Add DataNode maintenance states to Federation UI > -- > > Key: HDFS-17057 > URL: https://issues.apache.org/jira/browse/HDFS-17057 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Add DataNode maintenance states to Federation UI -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17053) Optimize method BlockInfoStriped#findSlot to reduce time complexity.
[
https://issues.apache.org/jira/browse/HDFS-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17053:
--
Component/s: hdfs
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Optimize method BlockInfoStriped#findSlot to reduce time complexity.
>
>
> Key: HDFS-17053
> URL: https://issues.apache.org/jira/browse/HDFS-17053
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Currently, in method findSlot. there exists codes snippet below:
> {code:java}
> for (; i < getCapacity(); i++) {
> if (getStorageInfo(i) == null) {
> return i;
> }
> } {code}
> it will compute （triplets.length / 3;) every iteration, I think this can be
> optimized.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17052) Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack.
[ https://issues.apache.org/jira/browse/HDFS-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17052: -- Target Version/s: 3.4.0 > Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed > when no enough Rack. > --- > > Key: HDFS-17052 > URL: https://issues.apache.org/jira/browse/HDFS-17052 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: failed reconstruction ec in same rack-1.png, write ec in > same rack.png > > > When writing EC data, if the number of racks matching the storageType is > insufficient, more than one block are allowed to be written to the same rack > !write ec in same rack.png|width=962,height=604! > > > > However, during EC block recovery, it is not possible to recover on the same > rack, which deviates from the expected behavior. > !failed reconstruction ec in same rack-1.png|width=946,height=413! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17045) File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted.
[
https://issues.apache.org/jira/browse/HDFS-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17045:
--
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> File renamed from a snapshottable dir to a non-snapshottable dir cannot be
> deleted.
> ---
>
> Key: HDFS-17045
> URL: https://issues.apache.org/jira/browse/HDFS-17045
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, snapshots
>Affects Versions: 3.4.0
>Reporter: Tsz-wo Sze
>Assignee: Tsz-wo Sze
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> HDFS-16972 added a
> [shouldDestroy|https://github.com/szetszwo/hadoop/blob/331e075115b4a35574622318b26f6d4731658d57/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java#L834-L845]
> method which caused the following bug.
> h3. Background:
> - When {{FileSystem.rename(src, dst)}} from a snapshottable dir (src) to a
> snapshottable dir (dst), dstSnapshotId is set to the latest snapshot at dst.
> As a result, dst is NOT in dstSnapshotId because dstSnapshotId was already
> taken before rename.
> - snapshotToBeDeleted is the snapshot id of the current operation if the
> operation is {{{}FileSystem.deleteSnapshot{}}}. Otherwise,
> snapshotToBeDeleted is set to CURRENT_STATE_ID.
> - If (snapshotToBeDeleted > dstSnapshotId), dst is in snapshotToBeDeleted.
> The shouldDestroy method returns true to continue deletion.
> - If (snapshotToBeDeleted <= dstSnapshotId), dst must not be in
> snapshotToBeDeleted. The shouldDestroy method returns false to stop deletion.
> All the above are correct for renaming within snapshottable directories.
> h3. Bug:
> - If rename(src, dst) from a snapshottable dir (src) to a non-snapshottable
> dir (dst), dstSnapshotId becomes CURRENT_STATE_ID.
> - When {{FileSystem.delete(dst)}} happens, snapshotToBeDeleted is also set
> to CURRENT_STATE_ID.
> - In this case, snapshotToBeDeleted == dstSnapshotId, the shouldDestroy
> method will return false and it incorrectly stops the deletion.
> Not that this bug may cause fsimage corruption and quota miscalculation since
> some files can be partially deleted. Fortunately, this bug won't cause data
> loss.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17043) HttpFS implementation for getAllErasureCodingPolicies
[ https://issues.apache.org/jira/browse/HDFS-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17043: -- Target Version/s: 3.4.0 > HttpFS implementation for getAllErasureCodingPolicies > - > > Key: HDFS-17043 > URL: https://issues.apache.org/jira/browse/HDFS-17043 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HttpFS should support getAllErasureCodingPolicies API In order to be able to > retrieve all Erasure Coding Policies.. WebHdfs implementation available on > HDFS-17029. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17047) BlockManager#addStoredBlock should log storage id when AddBlockResult is REPLACED
[
https://issues.apache.org/jira/browse/HDFS-17047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17047:
--
Component/s: hdfs
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> BlockManager#addStoredBlock should log storage id when AddBlockResult is
> REPLACED
> -
>
> Key: HDFS-17047
> URL: https://issues.apache.org/jira/browse/HDFS-17047
> Project: Hadoop HDFS
> Issue Type: Wish
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Recently, we found some logs in active namenode frequently like belows:
>
> {code:java}
> 2023-06-12 05:34:09,821 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-12 05:34:09,892 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-12 11:34:07,932 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-12 11:34:08,027 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-12 17:34:08,742 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-12 17:34:08,813 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-12 23:34:09,752 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-12 23:34:09,812 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-13 05:34:08,065 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-13 05:34:08,144 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-13 11:34:08,638 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010
> 2023-06-13 11:34:08,681 WARN BlockStateChange: BLOCK* addStoredBlock: block
> blk_-9223372036614126544_57136788 moved to storageType DISK on node
> datanode1:50010{code}
>
>
> All logs have the same ec block id : blk_-9223372036614126544_57136788 and
> printed every 6 hours(FBR interval of our cluster).
> To figure out what happened, I think we should also log storage id here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17045) File renamed from a snapshottable dir to a non-snapshottable dir cannot be deleted.
[
https://issues.apache.org/jira/browse/HDFS-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17045:
--
Hadoop Flags: Reviewed
> File renamed from a snapshottable dir to a non-snapshottable dir cannot be
> deleted.
> ---
>
> Key: HDFS-17045
> URL: https://issues.apache.org/jira/browse/HDFS-17045
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, snapshots
>Affects Versions: 3.4.0
>Reporter: Tsz-wo Sze
>Assignee: Tsz-wo Sze
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> HDFS-16972 added a
> [shouldDestroy|https://github.com/szetszwo/hadoop/blob/331e075115b4a35574622318b26f6d4731658d57/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java#L834-L845]
> method which caused the following bug.
> h3. Background:
> - When {{FileSystem.rename(src, dst)}} from a snapshottable dir (src) to a
> snapshottable dir (dst), dstSnapshotId is set to the latest snapshot at dst.
> As a result, dst is NOT in dstSnapshotId because dstSnapshotId was already
> taken before rename.
> - snapshotToBeDeleted is the snapshot id of the current operation if the
> operation is {{{}FileSystem.deleteSnapshot{}}}. Otherwise,
> snapshotToBeDeleted is set to CURRENT_STATE_ID.
> - If (snapshotToBeDeleted > dstSnapshotId), dst is in snapshotToBeDeleted.
> The shouldDestroy method returns true to continue deletion.
> - If (snapshotToBeDeleted <= dstSnapshotId), dst must not be in
> snapshotToBeDeleted. The shouldDestroy method returns false to stop deletion.
> All the above are correct for renaming within snapshottable directories.
> h3. Bug:
> - If rename(src, dst) from a snapshottable dir (src) to a non-snapshottable
> dir (dst), dstSnapshotId becomes CURRENT_STATE_ID.
> - When {{FileSystem.delete(dst)}} happens, snapshotToBeDeleted is also set
> to CURRENT_STATE_ID.
> - In this case, snapshotToBeDeleted == dstSnapshotId, the shouldDestroy
> method will return false and it incorrectly stops the deletion.
> Not that this bug may cause fsimage corruption and quota miscalculation since
> some files can be partially deleted. Fortunately, this bug won't cause data
> loss.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17044) Set size of non-exist block to NO_ACK when process FBR or IBR to avoid useless report from DataNode
[ https://issues.apache.org/jira/browse/HDFS-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17044: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Set size of non-exist block to NO_ACK when process FBR or IBR to avoid > useless report from DataNode > --- > > Key: HDFS-17044 > URL: https://issues.apache.org/jira/browse/HDFS-17044 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When NameNode processes DataNode increment or full block report, if block is > not in the blocks map, it will be added to invalidate and the replica should > be removed from the data-node, and the block size should be to set NO_ACK, > should be reduce some useless DataNode block reports. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17041) RBF: Fix putAll impl for mysql and file based state stores
[ https://issues.apache.org/jira/browse/HDFS-17041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17041: -- Component/s: rbf Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > RBF: Fix putAll impl for mysql and file based state stores > -- > > Key: HDFS-17041 > URL: https://issues.apache.org/jira/browse/HDFS-17041 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Only zookeeper based state store allows all records to be inserted even > though only few of them already exists and "errorIfExists" is true, however > file/fs as well as mysql based putAll fails the whole putAll operation > immediately after encountering single record that already exists in the > records and "errorIfExists" is provided true (which is the case while > inserting records for the first time). > For all implementations, we should allow inserts of the records that do not > already exist and report any record as failure that already exists, rather > than failing the whole operation and not trying to insert valid records. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17037) Consider nonDfsUsed when running balancer
[ https://issues.apache.org/jira/browse/HDFS-17037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17037: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Consider nonDfsUsed when running balancer > - > > Key: HDFS-17037 > URL: https://issues.apache.org/jira/browse/HDFS-17037 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When we run balancer with `BalancingPolicy.Node` policy, our goal is to make > each datanode storage balanced. But in the current implementation, the > balancer doesn't account for storage used by non-dfs on the datanodes, which > can make the situation worse for datanodes that are already strained on > storage. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17031) RBF: Reduce repeated code in RouterRpcServer
[
https://issues.apache.org/jira/browse/HDFS-17031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17031:
--
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> RBF: Reduce repeated code in RouterRpcServer
>
>
> Key: HDFS-17031
> URL: https://issues.apache.org/jira/browse/HDFS-17031
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Chengwei Wang
>Assignee: Chengwei Wang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Reduce repeated codes :
>
> {code:java}
> if (subclusterResolver instanceof MountTableResolver) {
> try {
> MountTableResolver mountTable = (MountTableResolver)subclusterResolver;
> MountTable entry = mountTable.getMountPoint(path);
> // check logic
> } catch (IOException e) {
> LOG.error("Cannot get mount point", e);
> }
> }
> return false; {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17035) FsVolumeImpl#getActualNonDfsUsed may return negative value
[ https://issues.apache.org/jira/browse/HDFS-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17035: -- Target Version/s: 3.4.0 > FsVolumeImpl#getActualNonDfsUsed may return negative value > -- > > Key: HDFS-17035 > URL: https://issues.apache.org/jira/browse/HDFS-17035 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17027) RBF: Add supports for observer.auto-msync-period when using routers
[ https://issues.apache.org/jira/browse/HDFS-17027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17027: -- Affects Version/s: 3.4.0 > RBF: Add supports for observer.auto-msync-period when using routers > --- > > Key: HDFS-17027 > URL: https://issues.apache.org/jira/browse/HDFS-17027 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > None-RBF clients that use observer reads have the option to set > *dfs.client.failover.observer.auto-msync-period.* . This config > makes the client automatically do an msync, allowing clients to use the > observer reads feature without any code change. > To use observer reads with RBF, clients set > *dfs.client.rbf.observer.read.enable*. The way this flag is implemented does > not allow clients to use the *auto-msync-period* config. So with RBF, clients > either have to > # Not use observer reads > # Use observer reads with the risk of stale reads > # Make code changes to explicitly call msync. > We should add support for > *dfs.client.failover.observer.auto-msync-period.*. This can be > done by adding a ProxyProvider, in a similar manner to the > ObserverReadProxyProvider. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17027) RBF: Add supports for observer.auto-msync-period when using routers
[ https://issues.apache.org/jira/browse/HDFS-17027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17027: -- Component/s: rbf Target Version/s: 3.4.0 > RBF: Add supports for observer.auto-msync-period when using routers > --- > > Key: HDFS-17027 > URL: https://issues.apache.org/jira/browse/HDFS-17027 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Affects Versions: 3.4.0 >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > None-RBF clients that use observer reads have the option to set > *dfs.client.failover.observer.auto-msync-period.* . This config > makes the client automatically do an msync, allowing clients to use the > observer reads feature without any code change. > To use observer reads with RBF, clients set > *dfs.client.rbf.observer.read.enable*. The way this flag is implemented does > not allow clients to use the *auto-msync-period* config. So with RBF, clients > either have to > # Not use observer reads > # Use observer reads with the risk of stale reads > # Make code changes to explicitly call msync. > We should add support for > *dfs.client.failover.observer.auto-msync-period.*. This can be > done by adding a ProxyProvider, in a similar manner to the > ObserverReadProxyProvider. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17026) RBF: NamenodeHeartbeatService should update JMX report with configurable frequency
[
https://issues.apache.org/jira/browse/HDFS-17026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17026:
--
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> RBF: NamenodeHeartbeatService should update JMX report with configurable
> frequency
> --
>
> Key: HDFS-17026
> URL: https://issues.apache.org/jira/browse/HDFS-17026
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Hector Sandoval Chaverri
>Assignee: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-17026-branch-3.3.patch
>
>
> The NamenodeHeartbeatService currently calls each of the Namenode's JMX
> endpoint every time it wakes up (default value is every 5 seconds).
> In a cluster with 40 routers, we have observed service degradation on some of
> the Namenodes, since the JMX request obtains Datanode status and blocks
> other RPC requests. However, JMX report data doesn't seem to be used for
> critical paths on the routers.
> We should configure the NamenodeHeartbeatService so it updates the JMX
> reports on a slower frequency than the Namenode states or to disable the
> reports completely.
> The class calls out the JMX request being optional even though there is no
> implementation to turn it off:
> {noformat}
> // Read the stats from JMX (optional)
> updateJMXParameters(webAddress, report);{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17029) Support getECPolices API in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-17029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17029: -- Target Version/s: 3.4.0 > Support getECPolices API in WebHDFS > --- > > Key: HDFS-17029 > URL: https://issues.apache.org/jira/browse/HDFS-17029 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-05-29-23-55-09-224.png > > > WebHDFS should support getEcPolicies: > !image-2023-05-29-23-55-09-224.png|width=817,height=234! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17024) Potential data race introduced by HDFS-15865
[
https://issues.apache.org/jira/browse/HDFS-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17024:
--
Hadoop Flags: Reviewed
> Potential data race introduced by HDFS-15865
>
>
> Key: HDFS-17024
> URL: https://issues.apache.org/jira/browse/HDFS-17024
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 3.3.1
>Reporter: Wei-Chiu Chuang
>Assignee: Segawa Hiroaki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> After HDFS-15865, we found client aborted due to an NPE.
> {noformat}
> 2023-04-10 16:07:43,409 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server kqhdp36,16020,1678077077562: Replay of WAL required. Forcing server
> shutdown *
> org.apache.hadoop.hbase.DroppedSnapshotException: region: WAFER_ALL,16|CM
> RIE.MA1|CP1114561.18|PROC|,1625899466315.0fbdf0f1810efa9e68af831247e6555f.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2870)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2539)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2511)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2401)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:613)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:582)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:362)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DataStreamer.waitForAckedSeqno(DataStreamer.java:880)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:781)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:898)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:850)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:76)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:105)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishClose(HFileWriterImpl.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:687)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:393)
> at
> org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:78)
> at
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1047)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2349)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2806)
> {noformat}
> This is only possible if a data race happened. File this jira to improve the
> data and eliminate the data race.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17022) Fix the exception message to print the Identifier pattern
[
https://issues.apache.org/jira/browse/HDFS-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17022:
--
Target Version/s: 3.3.6, 3.4.0
Affects Version/s: 3.3.6
3.4.0
> Fix the exception message to print the Identifier pattern
> -
>
> Key: HDFS-17022
> URL: https://issues.apache.org/jira/browse/HDFS-17022
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.4.0, 3.3.6
>Reporter: Nishtha Shah
>Assignee: Nishtha Shah
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> In case of an incorrect string passed as value, it would throw an exception,
> but the message doesn't print the identifier pattern.
> {code:java}
> java.lang.IllegalArgumentException: [] = [[a] must be {2}{code}
> instead of
> {code:java}
> java.lang.IllegalArgumentException: [] = [[a] must be
> [a-zA-Z_][a-zA-Z0-9_\-]*{code}
> Ref to original discussion:
> https://github.com/apache/hadoop/pull/5669#discussion_r1198937053
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17020) RBF: mount table addAll should print failed records in std error
[ https://issues.apache.org/jira/browse/HDFS-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17020: -- Affects Version/s: 3.4.0 > RBF: mount table addAll should print failed records in std error > > > Key: HDFS-17020 > URL: https://issues.apache.org/jira/browse/HDFS-17020 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Now that state store putAll supports returning failed records keys, addAll > command for mount entries should also support printing failed records in the > standard error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17022) Fix the exception message to print the Identifier pattern
[
https://issues.apache.org/jira/browse/HDFS-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17022:
--
Component/s: httpfs
> Fix the exception message to print the Identifier pattern
> -
>
> Key: HDFS-17022
> URL: https://issues.apache.org/jira/browse/HDFS-17022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: httpfs
>Affects Versions: 3.4.0, 3.3.6
>Reporter: Nishtha Shah
>Assignee: Nishtha Shah
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> In case of an incorrect string passed as value, it would throw an exception,
> but the message doesn't print the identifier pattern.
> {code:java}
> java.lang.IllegalArgumentException: [] = [[a] must be {2}{code}
> instead of
> {code:java}
> java.lang.IllegalArgumentException: [] = [[a] must be
> [a-zA-Z_][a-zA-Z0-9_\-]*{code}
> Ref to original discussion:
> https://github.com/apache/hadoop/pull/5669#discussion_r1198937053
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17020) RBF: mount table addAll should print failed records in std error
[ https://issues.apache.org/jira/browse/HDFS-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17020: -- Component/s: rbf > RBF: mount table addAll should print failed records in std error > > > Key: HDFS-17020 > URL: https://issues.apache.org/jira/browse/HDFS-17020 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Now that state store putAll supports returning failed records keys, addAll > command for mount entries should also support printing failed records in the > standard error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17018) Improve dfsclient log format
[ https://issues.apache.org/jira/browse/HDFS-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17018: -- Target Version/s: 3.4.0 > Improve dfsclient log format > > > Key: HDFS-17018 > URL: https://issues.apache.org/jira/browse/HDFS-17018 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsclient >Affects Versions: 3.3.4 >Reporter: Xianming Lei >Assignee: Xianming Lei >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Modify the log format. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17019) Optimize the logic for reconfigure slow peer enable for Namenode
[ https://issues.apache.org/jira/browse/HDFS-17019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17019: -- Component/s: namanode Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Optimize the logic for reconfigure slow peer enable for Namenode > - > > Key: HDFS-17019 > URL: https://issues.apache.org/jira/browse/HDFS-17019 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The logic of Reconfigure slow peer enable for Namenode requires the following > optimizations: > 1.Make SlowPeerTracker slowPeerTracker volatile. > 2.When starting the NameNode, if the parameter > dfs.datanode.peer.stats.enabled is set to false, > DatanodeManager#startSlowPeerCollector() will not call, as a result the Slow > peers collection thread 'slowPeerCollectorDaemon' will not be started . > If the parameter dfs.datanode.peer.stats.enabled is dynamically refreshed to > true, the current logic will not call > DatanodeManager#startSlowPeerCollector(), which to thread > 'slowPeerCollectorDaemon' not started as expected, so we will optimize here -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17020) RBF: mount table addAll should print failed records in std error
[ https://issues.apache.org/jira/browse/HDFS-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17020: -- Target Version/s: 3.4.0 > RBF: mount table addAll should print failed records in std error > > > Key: HDFS-17020 > URL: https://issues.apache.org/jira/browse/HDFS-17020 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Now that state store putAll supports returning failed records keys, addAll > command for mount entries should also support printing failed records in the > standard error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17019) Optimize the logic for reconfigure slow peer enable for Namenode
[ https://issues.apache.org/jira/browse/HDFS-17019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17019: -- Hadoop Flags: Reviewed > Optimize the logic for reconfigure slow peer enable for Namenode > - > > Key: HDFS-17019 > URL: https://issues.apache.org/jira/browse/HDFS-17019 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Affects Versions: 3.4.0 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The logic of Reconfigure slow peer enable for Namenode requires the following > optimizations: > 1.Make SlowPeerTracker slowPeerTracker volatile. > 2.When starting the NameNode, if the parameter > dfs.datanode.peer.stats.enabled is set to false, > DatanodeManager#startSlowPeerCollector() will not call, as a result the Slow > peers collection thread 'slowPeerCollectorDaemon' will not be started . > If the parameter dfs.datanode.peer.stats.enabled is dynamically refreshed to > true, the current logic will not call > DatanodeManager#startSlowPeerCollector(), which to thread > 'slowPeerCollectorDaemon' not started as expected, so we will optimize here -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17017) Fix the issue of arguments number limit in report command in DFSAdmin.
[ https://issues.apache.org/jira/browse/HDFS-17017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17017: -- Component/s: dfsadmin Target Version/s: 3.3.6, 3.4.0 Affects Version/s: 3.3.6 3.4.0 > Fix the issue of arguments number limit in report command in DFSAdmin. > -- > > Key: HDFS-17017 > URL: https://issues.apache.org/jira/browse/HDFS-17017 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsadmin >Affects Versions: 3.4.0, 3.3.6 >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Currently, the DFSAdmin report command should support a maximum number of > arguments of 7， such as : > hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning] > [-enteringmaintenance] [-inmaintenance] [-slownodes] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17012) Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT
[ https://issues.apache.org/jira/browse/HDFS-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17012: -- Labels: pull-requests-available (was: ) > Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT > > > Key: HDFS-17012 > URL: https://issues.apache.org/jira/browse/HDFS-17012 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs >Affects Versions: 3.3.4 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-requests-available > Fix For: 3.4.0 > > Attachments: screenshot-1.png > > > In DFSConfigKeys, DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT doesn't seem to have > been used anywhere, this is a redundant option and we should remove it. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17014) HttpFS Add Support getStatus API
[
https://issues.apache.org/jira/browse/HDFS-17014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17014:
--
Target Version/s: 3.4.0
> HttpFS Add Support getStatus API
>
>
> Key: HDFS-17014
> URL: https://issues.apache.org/jira/browse/HDFS-17014
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 3.4.0
>Reporter: Hualong Zhang
>Assignee: Hualong Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: image-2023-05-24-21-58-28-674.png
>
>
> We should ensure that *WebHDFS* remains synchronized with {*}HttpFS{*}, as
> the former has already implemented the *getStatus* interface.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17015) Typos in HDFS Documents
[ https://issues.apache.org/jira/browse/HDFS-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17015: -- Target Version/s: 3.4.0 > Typos in HDFS Documents > --- > > Key: HDFS-17015 > URL: https://issues.apache.org/jira/browse/HDFS-17015 > Project: Hadoop HDFS > Issue Type: Improvement > Components: configuration >Affects Versions: 3.3.5 >Reporter: Liang Yan >Assignee: Liang Yan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > There are some typos in HDFS documents. I will submit PR to fix these typos. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17012) Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT
[ https://issues.apache.org/jira/browse/HDFS-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17012: -- Target Version/s: 3.4.0 > Remove unused DFSConfigKeys#DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT > > > Key: HDFS-17012 > URL: https://issues.apache.org/jira/browse/HDFS-17012 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, hdfs >Affects Versions: 3.3.4 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Minor > Labels: pull-requests-available > Fix For: 3.4.0 > > Attachments: screenshot-1.png > > > In DFSConfigKeys, DFS_DATANODE_PMEM_CACHE_DIRS_DEFAULT doesn't seem to have > been used anywhere, this is a redundant option and we should remove it. > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17011) Fix the metric of "HttpPort" at DataNodeInfo
[ https://issues.apache.org/jira/browse/HDFS-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17011: -- Hadoop Flags: Reviewed Target Version/s: 3.3.6, 3.4.0 Affects Version/s: 3.3.6 3.4.0 > Fix the metric of "HttpPort" at DataNodeInfo > -- > > Key: HDFS-17011 > URL: https://issues.apache.org/jira/browse/HDFS-17011 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.4.0, 3.3.6 >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > Attachments: after.png, before.png > > > Now, the "HttpPort" metric was getting from the conf `dfs.datanode.info.port` > but the conf seem to be useless, httpPort already named infoPort and was > assigned (#Line1373) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17011) Fix the metric of "HttpPort" at DataNodeInfo
[ https://issues.apache.org/jira/browse/HDFS-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17011: -- Labels: pull-request-available (was: ) > Fix the metric of "HttpPort" at DataNodeInfo > -- > > Key: HDFS-17011 > URL: https://issues.apache.org/jira/browse/HDFS-17011 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > Attachments: after.png, before.png > > > Now, the "HttpPort" metric was getting from the conf `dfs.datanode.info.port` > but the conf seem to be useless, httpPort already named infoPort and was > assigned (#Line1373) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17011) Fix the metric of "HttpPort" at DataNodeInfo
[ https://issues.apache.org/jira/browse/HDFS-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17011: -- Component/s: datanode > Fix the metric of "HttpPort" at DataNodeInfo > -- > > Key: HDFS-17011 > URL: https://issues.apache.org/jira/browse/HDFS-17011 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > Attachments: after.png, before.png > > > Now, the "HttpPort" metric was getting from the conf `dfs.datanode.info.port` > but the conf seem to be useless, httpPort already named infoPort and was > assigned (#Line1373) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17010) Add a subtree test to TestSnapshotDiffReport
[ https://issues.apache.org/jira/browse/HDFS-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17010: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Add a subtree test to TestSnapshotDiffReport > > > Key: HDFS-17010 > URL: https://issues.apache.org/jira/browse/HDFS-17010 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 3.4.0 >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Minor > Fix For: 3.4.0 > > > The JIRA is to add a test for running SnapshotDiffReport over subtrees of a > snapshottable directory. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17009) RBF: state store putAll should also return failed records
[ https://issues.apache.org/jira/browse/HDFS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17009: -- Affects Version/s: 3.4.0 > RBF: state store putAll should also return failed records > - > > Key: HDFS-17009 > URL: https://issues.apache.org/jira/browse/HDFS-17009 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > State store implementations allow adding/updating multiple records using > putAll. The implementation returns whether all records were successfully > added or updated. We should also allow the implementation to return which > records failed to get updated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17009) RBF: state store putAll should also return failed records
[ https://issues.apache.org/jira/browse/HDFS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17009: -- Target Version/s: 3.4.0 > RBF: state store putAll should also return failed records > - > > Key: HDFS-17009 > URL: https://issues.apache.org/jira/browse/HDFS-17009 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > State store implementations allow adding/updating multiple records using > putAll. The implementation returns whether all records were successfully > added or updated. We should also allow the implementation to return which > records failed to get updated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17003) Erasure Coding: invalidate wrong block after reporting bad blocks from datanode
[
https://issues.apache.org/jira/browse/HDFS-17003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17003:
--
Affects Version/s: 3.3.6
3.4.0
> Erasure Coding: invalidate wrong block after reporting bad blocks from
> datanode
> ---
>
> Key: HDFS-17003
> URL: https://issues.apache.org/jira/browse/HDFS-17003
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.4.0, 3.3.6
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> After receiving reportBadBlocks RPC from datanode, NameNode compute wrong
> block to invalidate. It is a dangerous behaviour and may cause data loss.
> Some logs in our production as below:
>
> NameNode log:
> {code:java}
> 2023-05-08 21:23:49,112 INFO org.apache.hadoop.hdfs.StateChange: *DIR*
> reportBadBlocks for block:
> BP-932824627--1680179358678:blk_-9223372036848404320_1471186 on datanode:
> datanode1:50010
> 2023-05-08 21:23:49,183 INFO org.apache.hadoop.hdfs.StateChange: *DIR*
> reportBadBlocks for block:
> BP-932824627--1680179358678:blk_-9223372036848404319_1471186 on datanode:
> datanode2:50010{code}
> datanode1 log:
> {code:java}
> 2023-05-08 21:23:49,088 WARN
> org.apache.hadoop.hdfs.server.datanode.VolumeScanner: Reporting bad
> BP-932824627--1680179358678:blk_-9223372036848404320_1471186 on
> /data7/hadoop/hdfs/datanode
> 2023-05-08 21:24:00,509 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Failed
> to delete replica blk_-9223372036848404319_1471186: ReplicaInfo not
> found.{code}
>
> This phenomenon can be reproduced.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17009) RBF: state store putAll should also return failed records
[ https://issues.apache.org/jira/browse/HDFS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17009: -- Labels: pull-request-available (was: ) > RBF: state store putAll should also return failed records > - > > Key: HDFS-17009 > URL: https://issues.apache.org/jira/browse/HDFS-17009 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > State store implementations allow adding/updating multiple records using > putAll. The implementation returns whether all records were successfully > added or updated. We should also allow the implementation to return which > records failed to get updated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17008) Fix RBF JDK 11 javadoc warnings
[ https://issues.apache.org/jira/browse/HDFS-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17008: -- Component/s: rbf Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix RBF JDK 11 javadoc warnings > --- > > Key: HDFS-17008 > URL: https://issues.apache.org/jira/browse/HDFS-17008 > Project: Hadoop HDFS > Issue Type: Task > Components: rbf >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HDFS-16978 excluded proto packages from maven-javadoc-plugin for rbf, hence > now we have JDK 11 javadoc warnings (e.g. > [here|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5554/14/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1.txt]). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17009) RBF: state store putAll should also return failed records
[ https://issues.apache.org/jira/browse/HDFS-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17009: -- Component/s: rbf > RBF: state store putAll should also return failed records > - > > Key: HDFS-17009 > URL: https://issues.apache.org/jira/browse/HDFS-17009 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > State store implementations allow adding/updating multiple records using > putAll. The implementation returns whether all records were successfully > added or updated. We should also allow the implementation to return which > records failed to get updated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17001) Support getStatus API in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-17001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-17001: -- Target Version/s: 3.4.0 > Support getStatus API in WebHDFS > > > Key: HDFS-17001 > URL: https://issues.apache.org/jira/browse/HDFS-17001 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-05-08-14-34-51-873.png > > > WebHDFS should support getStatus: > !image-2023-05-08-14-34-51-873.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17000) Potential infinite loop in TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline
[
https://issues.apache.org/jira/browse/HDFS-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17000:
--
Component/s: test
> Potential infinite loop in
> TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline
>
>
> Key: HDFS-17000
> URL: https://issues.apache.org/jira/browse/HDFS-17000
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Marcono1234
>Assignee: Marcono1234
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The method
> {{TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline}}
> contains the following line:
> {code}
> for (int i = 0; i < Long.MAX_VALUE; i++) {
> {code}
> [GitHub source
> link|https://github.com/apache/hadoop/blob/4ee92efb73a90ae7f909e96de242d216ad6878b2/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamUpdatePipeline.java#L48]
> Because {{i}} is an {{int}} the condition {{i < Long.MAX_VALUE}} will always
> be true and {{i}} will simply overflow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16999) Fix wrong use of processFirstBlockReport()
[ https://issues.apache.org/jira/browse/HDFS-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16999: -- Component/s: namanode Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix wrong use of processFirstBlockReport() > -- > > Key: HDFS-16999 > URL: https://issues.apache.org/jira/browse/HDFS-16999 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > `processFirstBlockReport()` is used to process first block report from > datanode. It does not calculating `toRemove` list because it believes that > there is no metadata about the datanode in the namenode. However, If a > datanode is re registered after restarting, its `blockReportCount` will be > updated to 0. That is to say, the first block report after a datanode > restarts will be processed by `processFirstBlockReport()`. This is > unreasonable because the metadata of the datanode already exists in namenode > at this time, and if redundant replica metadata is not removed in time, the > blocks with insufficient replicas cannot be reconstruct in time, which > increases the risk of missing block. In summary, `processFirstBlockReport()` > should only be used when the namenode restarts, not when the datanode > restarts. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17000) Potential infinite loop in TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline
[
https://issues.apache.org/jira/browse/HDFS-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-17000:
--
Target Version/s: 3.4.0
Affects Version/s: 3.4.0
> Potential infinite loop in
> TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline
>
>
> Key: HDFS-17000
> URL: https://issues.apache.org/jira/browse/HDFS-17000
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0
>Reporter: Marcono1234
>Assignee: Marcono1234
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> The method
> {{TestDFSStripedOutputStreamUpdatePipeline.testDFSStripedOutputStreamUpdatePipeline}}
> contains the following line:
> {code}
> for (int i = 0; i < Long.MAX_VALUE; i++) {
> {code}
> [GitHub source
> link|https://github.com/apache/hadoop/blob/4ee92efb73a90ae7f909e96de242d216ad6878b2/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamUpdatePipeline.java#L48]
> Because {{i}} is an {{int}} the condition {{i < Long.MAX_VALUE}} will always
> be true and {{i}} will simply overflow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16998) RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity
[ https://issues.apache.org/jira/browse/HDFS-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16998: -- Component/s: rbf Target Version/s: 3.4.0 > RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity > -- > > Key: HDFS-16998 > URL: https://issues.apache.org/jira/browse/HDFS-16998 > Project: Hadoop HDFS > Issue Type: Task > Components: rbf >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16997) Set the locale to avoid printing useless logs in BlockSender
[ https://issues.apache.org/jira/browse/HDFS-16997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16997: -- Component/s: block placement Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Set the locale to avoid printing useless logs in BlockSender > > > Key: HDFS-16997 > URL: https://issues.apache.org/jira/browse/HDFS-16997 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > In our production environment, if the hadoop process is started in a > non-English environment, many unexpected error logs will be printed. The > following is the error message printed by datanode. > ``` > 2023-05-01 09:10:50,299 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,299 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258) > at java.lang.Thread.run(Thread.java:745) > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,302 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258) > at java.lang.Thread.run(Thread.java:745) > 2023-05-01 09:10:50,303 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,303 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChann
[jira] [Updated] (HDFS-16999) Fix wrong use of processFirstBlockReport()
[ https://issues.apache.org/jira/browse/HDFS-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16999: -- Component/s: block placement (was: namanode) > Fix wrong use of processFirstBlockReport() > -- > > Key: HDFS-16999 > URL: https://issues.apache.org/jira/browse/HDFS-16999 > Project: Hadoop HDFS > Issue Type: Bug > Components: block placement >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > `processFirstBlockReport()` is used to process first block report from > datanode. It does not calculating `toRemove` list because it believes that > there is no metadata about the datanode in the namenode. However, If a > datanode is re registered after restarting, its `blockReportCount` will be > updated to 0. That is to say, the first block report after a datanode > restarts will be processed by `processFirstBlockReport()`. This is > unreasonable because the metadata of the datanode already exists in namenode > at this time, and if redundant replica metadata is not removed in time, the > blocks with insufficient replicas cannot be reconstruct in time, which > increases the risk of missing block. In summary, `processFirstBlockReport()` > should only be used when the namenode restarts, not when the datanode > restarts. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16998) RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity
[ https://issues.apache.org/jira/browse/HDFS-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16998: -- Affects Version/s: 3.4.0 > RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity > -- > > Key: HDFS-16998 > URL: https://issues.apache.org/jira/browse/HDFS-16998 > Project: Hadoop HDFS > Issue Type: Task >Affects Versions: 3.4.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16995) Remove unused parameters at NameNodeHttpServer#initWebHdfs
[ https://issues.apache.org/jira/browse/HDFS-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16995: -- Component/s: webhdfs Hadoop Flags: Reviewed Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Remove unused parameters at NameNodeHttpServer#initWebHdfs > -- > > Key: HDFS-16995 > URL: https://issues.apache.org/jira/browse/HDFS-16995 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.4.0 >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16986) EC: Fix locationBudget in getListing()
[ https://issues.apache.org/jira/browse/HDFS-16986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16986: -- Component/s: erasure-coding Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > EC: Fix locationBudget in getListing() > -- > > Key: HDFS-16986 > URL: https://issues.apache.org/jira/browse/HDFS-16986 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.4.0 >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The current `locationBudget` is estimated using the `block_replication` in > `FileStatus`, which is unreasonable on EC files, because it will count the > number of locations of a EC block as 1. We should consider > ErasureCodingPolicy of the files to keep the meaning of `locationBudget` > consistent. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16990) HttpFS Add Support getFileLinkStatus API
[ https://issues.apache.org/jira/browse/HDFS-16990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16990: -- Target Version/s: 3.4.0 > HttpFS Add Support getFileLinkStatus API > > > Key: HDFS-16990 > URL: https://issues.apache.org/jira/browse/HDFS-16990 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HttpFS should implement the *getFileLinkStatus* API already implemented in > WebHDFS. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16988) Improve NameServices info at JournalNode web UI
[ https://issues.apache.org/jira/browse/HDFS-16988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16988: -- Component/s: journal-node ui Target Version/s: 3.3.6, 3.4.0 Affects Version/s: 3.3.6 3.4.0 > Improve NameServices info at JournalNode web UI > --- > > Key: HDFS-16988 > URL: https://issues.apache.org/jira/browse/HDFS-16988 > Project: Hadoop HDFS > Issue Type: Improvement > Components: journal-node, ui >Affects Versions: 3.4.0, 3.3.6 >Reporter: Zhaohui Wang >Assignee: Zhaohui Wang >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > Attachments: Before.png, after.png > > > If the NameServices is named xxx-abc-edg, only xxx will be displayed on the > JN web UI. > If NS1 is named xxx-abc-edg, NS2 is named xxx-lmn-xyz. Show both NS as xxx on > the JN web UI. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16985) Fix data missing issue when delete local block file.
[ https://issues.apache.org/jira/browse/HDFS-16985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16985: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Fix data missing issue when delete local block file. > > > Key: HDFS-16985 > URL: https://issues.apache.org/jira/browse/HDFS-16985 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.4.0 >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We encounterd several missing-block problem in our production cluster which > hdfs running on AWS EC2 + EBS. > The root cause： > # the block remains only 1 replication left and hasn't been reconstruction > # DN checks block file existing when BlockSender construction > # the EBS checking failed and throw FileNotFoundException (EBS may be in > fault condition) > # DN invalidateBlock and schedule block async deletion > # EBS already back to normal when DN do delete block > # the block file be delete permanently and can't be recovered -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16982) Use the right Quantiles Array for Inverse Quantiles snapshot
[ https://issues.apache.org/jira/browse/HDFS-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16982: -- Target Version/s: 3.4.0 Affects Version/s: 3.4.0 > Use the right Quantiles Array for Inverse Quantiles snapshot > - > > Key: HDFS-16982 > URL: https://issues.apache.org/jira/browse/HDFS-16982 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, metrics >Affects Versions: 3.4.0 >Reporter: Ravindra Dingankar >Assignee: Ravindra Dingankar >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > HDFS-16949 introduced InverseQuantiles. However during snapshot for Inverse > Quantiles we were still trying to access values from previous snapshot based > on the Quantile Array declared in MutableQuantiles. ( Quantile(.50, .050), > Quantile(.75, .025), Quantile(.90, .010), Quantile(.95, .005), Quantile(.99, > .001) ) > For InverseQuantiles we wont have these values ( except for Quantile(.50, > .050) ) thus except for 50 Percentile snapshot wont return any value for the > remaining quantiles. > Fix is to use the correct Quantiles Array to retrieve values during snapshot. > The new UTs verify this behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16983) Fix concat operation doesn't honor dfs.permissions.enabled
[
https://issues.apache.org/jira/browse/HDFS-16983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HDFS-16983:
--
Target Version/s: 3.4.0
> Fix concat operation doesn't honor dfs.permissions.enabled
> --
>
> Key: HDFS-16983
> URL: https://issues.apache.org/jira/browse/HDFS-16983
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> In concat RPC, it will call FSDirConcatOp::verifySrcFiles() to check the
> source files. In this function, it would make permission check for srcs.
> Whether do the permission check should be decided by dfs.permissions.enabled
> configuration. And the 'pc' parameter is always not null.
> So we should change 'if (pc != null)' to 'if (fsd.isPermissionEnabled())'.
> {code:java}
> // permission check for srcs
> if (pc != null) {
> fsd.checkPathAccess(pc, iip, FsAction.READ); // read the file
> fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16981) Support getFileLinkStatus API in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HDFS-16981: -- Target Version/s: 3.4.0 > Support getFileLinkStatus API in WebHDFS > > > Key: HDFS-16981 > URL: https://issues.apache.org/jira/browse/HDFS-16981 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.4.0 >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-04-13-23-41-51-380.png > > > WebHDFS should support getFileLinkStatus: > !image-2023-04-13-23-41-51-380.png|width=670,height=187! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
