[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead

2024-03-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Description: In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges from 5ms to

[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead

2024-03-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Issue Type: Wish (was: Task) > Selector.select in SocketIOWithTimeout.java has significant overhead >

[jira] [Commented] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead

2024-03-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828925#comment-17828925 ] qinyuren commented on HDFS-17434: - [~hexiaoqiao] [~tasanuma] [~zanderxu]  Please take a look. >

[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead

2024-03-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Attachment: image-2024-03-20-19-55-18-378.png > Selector.select in SocketIOWithTimeout.java has

[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead

2024-03-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Issue Type: Test (was: Wish) > Selector.select in SocketIOWithTimeout.java has significant overhead >

[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead

2024-03-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Description: In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges from 5ms to

[jira] [Created] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead

2024-03-20 Thread qinyuren (Jira)
qinyuren created HDFS-17434: --- Summary: Selector.select in SocketIOWithTimeout.java has significant overhead Key: HDFS-17434 URL: https://issues.apache.org/jira/browse/HDFS-17434 Project: Hadoop HDFS

[jira] [Comment Edited] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections

2023-11-06 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783445#comment-17783445 ] qinyuren edited comment on HDFS-15413 at 11/7/23 1:46 AM: -- We also encountered

[jira] [Comment Edited] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections

2023-11-06 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783445#comment-17783445 ] qinyuren edited comment on HDFS-15413 at 11/7/23 1:45 AM: -- We also encountered

[jira] [Commented] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections

2023-11-06 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783445#comment-17783445 ] qinyuren commented on HDFS-15413: - We also encountered datanode socket timeout problems in the process of

[jira] [Commented] (HDFS-17187) add the tool that getting the hdfs file path through block id

2023-09-12 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764141#comment-17764141 ] qinyuren commented on HDFS-17187: - *hdfs fsck -blockId * can get the hdfs file path. > add the tool that

[jira] [Updated] (HDFS-16784) replace readLock with writeLock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage

2022-09-28 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Summary: replace readLock with writeLock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage (was: use

[jira] [Updated] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage

2022-09-28 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Description: In #DatanodeAdminBackoffMonitor.scanDatanodeStorage, it uses a read lock to protect the

[jira] [Updated] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage

2022-09-28 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Attachment: image-2022-09-28-15-37-24-622.png > use write lock in

[jira] [Updated] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage

2022-09-28 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Description: In  > use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage >

[jira] [Created] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage

2022-09-28 Thread qinyuren (Jira)
qinyuren created HDFS-16784: --- Summary: use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage Key: HDFS-16784 URL: https://issues.apache.org/jira/browse/HDFS-16784 Project: Hadoop HDFS

[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode

2022-05-09 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16575: Description: The SPS may have misjudged in the following scenario: # Create a file with one block and

[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode

2022-05-09 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16575: Description: # create a file with one block and this block have 3 replication with

[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode

2022-05-09 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16575: Attachment: image-2022-05-10-11-21-31-987.png > [SPS]: Should use real replication num instead

[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode

2022-05-09 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16575: Attachment: image-2022-05-10-11-21-13-627.png > [SPS]: Should use real replication num instead

[jira] [Created] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode

2022-05-09 Thread qinyuren (Jira)
qinyuren created HDFS-16575: --- Summary: [SPS]: Should use real replication num instead getReplication from namenode Key: HDFS-16575 URL: https://issues.apache.org/jira/browse/HDFS-16575 Project: Hadoop HDFS

[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-19 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16544: Description: In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we found an EC file

[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-19 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16544: Description: In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we found an EC file

[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-15 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16544: Description: In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we found an EC file

[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-15 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16544: Description: In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we found a ec

[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-15 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16544: Description: In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538] , we found a ec

[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-15 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16544: Description: In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538]   {code:java}

[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-15 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16544: Description: In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538]   {code:java}

[jira] [Created] (HDFS-16544) EC decoding failed due to invalid buffer

2022-04-15 Thread qinyuren (Jira)
qinyuren created HDFS-16544: --- Summary: EC decoding failed due to invalid buffer Key: HDFS-16544 URL: https://issues.apache.org/jira/browse/HDFS-16544 Project: Hadoop HDFS Issue Type: Bug

[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs

2022-04-14 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16538: Description: Currently, we found this error if the #StripeReader.readStripe() have more than one block

[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs

2022-04-14 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16538: Description: Currently, we found this error if the #StripeReader.readStripe() have more than one block

[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs

2022-04-13 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16538: Description: Currently, we found this error if the #StripeReader.readStripe() have more than one block

[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs

2022-04-13 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16538: Description: We found this error if the #StripeReader.readStripe() have more than one block read failed.

[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs

2022-04-13 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16538: Description: We found this error if the #StripeReader.readStripe() have more than one block read failed.

[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs

2022-04-13 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16538: Description: We found this error if the #StripeReader.readStripe() have more than one block read failed.

[jira] [Created] (HDFS-16538) EC decoding failed due to not enough valid inputs

2022-04-13 Thread qinyuren (Jira)
qinyuren created HDFS-16538: --- Summary: EC decoding failed due to not enough valid inputs Key: HDFS-16538 URL: https://issues.apache.org/jira/browse/HDFS-16538 Project: Hadoop HDFS Issue Type: Bug

[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

2022-03-23 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Description: Currently, we ran SPS in our cluster and found this log. The SPSPathIdProcessor thread

[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as

[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as

[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as

[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode:

[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode:

[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode:

[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode:

[jira] [Created] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured

2022-03-21 Thread qinyuren (Jira)
qinyuren created HDFS-16514: --- Summary: Reduce the failover sleep time if multiple namenode are configured Key: HDFS-16514 URL: https://issues.apache.org/jira/browse/HDFS-16514 Project: Hadoop HDFS

[jira] [Created] (HDFS-16501) Print the exception when reporting a bad block

2022-03-10 Thread qinyuren (Jira)
qinyuren created HDFS-16501: --- Summary: Print the exception when reporting a bad block Key: HDFS-16501 URL: https://issues.apache.org/jira/browse/HDFS-16501 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process

2022-02-25 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16485: Description: When SPSPathIdProcessor thread call getNextSPSPath(), it get the pathId from namenode and

[jira] [Updated] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process

2022-02-25 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16485: Description: When SPSPathIdProcessor thread call getNextSPSPath(), it get the pathId from namenode and

[jira] [Updated] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process

2022-02-25 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16485: Description:   {code:java} public Long getNextPathId() { synchronized (pathsToBeTraveresed) { return

[jira] [Created] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process

2022-02-25 Thread qinyuren (Jira)
qinyuren created HDFS-16485: --- Summary: [SPS]: allow re-satisfy path after restarting sps process Key: HDFS-16485 URL: https://issues.apache.org/jira/browse/HDFS-16485 Project: Hadoop HDFS Issue

[jira] [Assigned] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-25 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren reassigned HDFS-16484: --- Assignee: qinyuren > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread >

[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-24 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Description: In SPSPathIdProcessor thread, if it get a inodeId which path does not exist, then the

[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-24 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Description: In SPSPathIdProcessor thread, if sps get a inodeId which path does not exist, then the

[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-24 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Description: !image-2022-02-25-14-35-42-255.png|width=682,height=195! > [SPS]: Fix an infinite loop bug in

[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-24 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Attachment: image-2022-02-25-14-35-42-255.png > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor

[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-24 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Summary: [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (was: [SPS]: fix an infinite loop

[jira] [Created] (HDFS-16484) fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-24 Thread qinyuren (Jira)
qinyuren created HDFS-16484: --- Summary: fix an infinite loop bug in SPSPathIdProcessor thread Key: HDFS-16484 URL: https://issues.apache.org/jira/browse/HDFS-16484 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-16484) [SPS]: fix an infinite loop bug in SPSPathIdProcessor thread

2022-02-24 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Summary: [SPS]: fix an infinite loop bug in SPSPathIdProcessor thread (was: fix an infinite loop bug in

[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-02-09 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Attachment: 20220209-120613.html > Namenode block report add yield to avoid holding write lock too long >

[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-02-08 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Attachment: (was: 20220209-120613.html) > Namenode block report add yield to avoid holding write lock

[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-02-08 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Attachment: 20220209-120613.html > Namenode block report add yield to avoid holding write lock too long >

[jira] [Commented] (HDFS-16423) balancer should not get blocks on stale storages

2022-01-26 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482862#comment-17482862 ] qinyuren commented on HDFS-16423: - Thanks [~sodonnell] for the advices. I will add a PR to indicate why

[jira] [Comment Edited] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-01-24 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17479811#comment-17479811 ] qinyuren edited comment on HDFS-16432 at 1/25/22, 6:29 AM: --- [~hexiaoqiao],

[jira] [Commented] (HDFS-16433) The synchronized lock of IncrementalBlockReportManager will slow down the speed of creating/deleting/finalizing/opening block

2022-01-23 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480786#comment-17480786 ] qinyuren commented on HDFS-16433: - I think that object.wait() will release the lock. > The synchronized

[jira] [Commented] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-01-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17479811#comment-17479811 ] qinyuren commented on HDFS-16432: - [~hexiaoqiao] [~tasanuma] Please take a look. > Namenode block report

[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-01-20 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Description: !image-2022-01-20-15-19-28-384.png|width=683,height=132! In our cluster, namenode block

[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-01-19 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Description: !image-2022-01-20-15-19-28-384.png|width=683,height=132! In our cluster, namenode block

[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-01-19 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Description: !image-2022-01-20-15-19-28-384.png|width=652,height=126! In our cluster, namenode block

[jira] [Created] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long

2022-01-19 Thread qinyuren (Jira)
qinyuren created HDFS-16432: --- Summary: Namenode block report add yield to avoid holding write lock too long Key: HDFS-16432 URL: https://issues.apache.org/jira/browse/HDFS-16432 Project: Hadoop HDFS

[jira] [Commented] (HDFS-16423) balancer should not get blocks on stale storages

2022-01-18 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17478312#comment-17478312 ] qinyuren commented on HDFS-16423: - Hi [~weichiu] , the problem we described in HDFS-16420 was not related

[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472419#comment-17472419 ] qinyuren commented on HDFS-16420: - Hi [~tasanuma] , we've merged these two PR into our code. It seems

[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472401#comment-17472401 ] qinyuren commented on HDFS-16420: - Hi [~ayushtkn] , we cherry-pick most of EC fixes which merged before

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Comment Edited] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471966#comment-17471966 ] qinyuren edited comment on HDFS-16420 at 1/10/22, 12:49 PM: Hi [~sodonnell],

[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471966#comment-17471966 ] qinyuren commented on HDFS-16420: - [~sodonnell] We cherry-pick most of EC fixes, especially those PR with

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Comment Edited] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471822#comment-17471822 ] qinyuren edited comment on HDFS-16420 at 1/10/22, 10:01 AM: [~hexiaoqiao] ,

[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471822#comment-17471822 ] qinyuren commented on HDFS-16420: - [~hexiaoqiao] [~ayushtkn] Please take a look at this problem. > ec +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16420: Description: We have a similar problem as HDFS-16297 described.  In our cluster, we used ec(6+3) +

[jira] [Created] (HDFS-16420) ec + balancer may cause missing block

2022-01-10 Thread qinyuren (Jira)
qinyuren created HDFS-16420: --- Summary: ec + balancer may cause missing block Key: HDFS-16420 URL: https://issues.apache.org/jira/browse/HDFS-16420 Project: Hadoop HDFS Issue Type: Bug

[jira] [Assigned] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport

2021-11-23 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren reassigned HDFS-16352: --- Assignee: qinyuren > return the real datanode numBlocks in #getDatanodeStorageReport >

[jira] [Updated] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport

2021-11-23 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Summary: return the real datanode numBlocks in #getDatanodeStorageReport (was: return the real datanode

[jira] [Updated] (HDFS-16352) return the real datanode blocknums in #getDatanodeStorageReport

2021-11-23 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Summary: return the real datanode blocknums in #getDatanodeStorageReport (was: fill the real datanode

[jira] [Updated] (HDFS-16352) fill the real datanode blocknums in #getDatanodeStorageReport

2021-11-23 Thread qinyuren (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Description: #getDatanodeStorageReport will return the array of DatanodeStorageReport which contains the

[jira] [Created] (HDFS-16352) fill the real datanode blocknums in #getDatanodeStorageReport

2021-11-23 Thread qinyuren (Jira)
qinyuren created HDFS-16352: --- Summary: fill the real datanode blocknums in #getDatanodeStorageReport Key: HDFS-16352 URL: https://issues.apache.org/jira/browse/HDFS-16352 Project: Hadoop HDFS

  1   2   >