[jira] [Commented] (HDFS-17323) Uncontrolled fsimage size due to snapshot diff meta for file deletions
[
https://issues.apache.org/jira/browse/HDFS-17323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804541#comment-17804541
]
Srinivasu Majeti commented on HDFS-17323:
-
Hi [~arp] / [~weichu] , Could you please take a look at this Jira?
> Uncontrolled fsimage size due to snapshot diff meta for file deletions
> --
>
> Key: HDFS-17323
> URL: https://issues.apache.org/jira/browse/HDFS-17323
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.1
>Reporter: Srinivasu Majeti
>Priority: Major
>
> We have seen quite a good number of customer cases w.r.t fsimage size
> increased drastically while storing snapshot meta for fileDiff entries. Here
> is an example fsimage meta storing entire inode info after deleting a file.
> I'm not sure about any restrictions on why the entire inode meta needs to be
> stored in fileDiff entry when there is no change w.r.t actual inode meta and
> it's just a delete file operation.
> The fileDiffEntry for the inode 1860467 seems redundant for a simple file
> delete operation.
> {code:java}
> 431860465DIRECTORYs31704197935903hdfs:supergroup:0755-1-1
> 441860465DIRECTORYs41704197951829hdfs:supergroup:0755-1-1
> 1860467FILEfile1317041979173151704197917031134217728hdfs:supergroup:06441074008442267653418
> 1860467file1043
> 186046721474836460
> 1860467143418file1317041979173151704197917031134217728hdfs:supergroup:06440
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17284) Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery
[ https://issues.apache.org/jira/browse/HDFS-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HDFS-17284: - Component/s: ec > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery > -- > > Key: HDFS-17284 > URL: https://issues.apache.org/jira/browse/HDFS-17284 > Project: Hadoop HDFS > Issue Type: Bug > Components: ec, namenode >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17290) HDFS: add client rpc backoff metrics due to disconnection from lowest priority queue
[ https://issues.apache.org/jira/browse/HDFS-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Yang updated HDFS-17290: Fix Version/s: 3.4.0 > HDFS: add client rpc backoff metrics due to disconnection from lowest > priority queue > > > Key: HDFS-17290 > URL: https://issues.apache.org/jira/browse/HDFS-17290 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.10.0, 3.4.0 >Reporter: Lei Yang >Assignee: Lei Yang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Clients are backoff when rpcs cannot be enqueued. However there are different > scenarios when backoff could happen. Currently there is no way to > differenciate whether a backoff happened due to lowest prio+disconnection or > queue overflow from higher priority queues when connection between client and > namenode remains open. Currently IPC server just emits a single metrics for > all the backoffs. > Example: > # Client are directly enqueued into lowest priority queue and backoff when > lowest queue is full. Client are expected to disconnect from namenode. > # Client are enqueued into non-lowest priority queue and overflowed all the > way down to lowest priority queue and back off. In this case, connection > between client and namenode remains open. > We would like to add metrics for #1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17290) HDFS: add client rpc backoff metrics due to disconnection from lowest priority queue
[ https://issues.apache.org/jira/browse/HDFS-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804483#comment-17804483 ] Lei Yang commented on HDFS-17290: - [~simbadzina] Thanks for reviewing and merging the PR. > HDFS: add client rpc backoff metrics due to disconnection from lowest > priority queue > > > Key: HDFS-17290 > URL: https://issues.apache.org/jira/browse/HDFS-17290 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.10.0, 3.4.0 >Reporter: Lei Yang >Assignee: Lei Yang >Priority: Major > Labels: pull-request-available > > Clients are backoff when rpcs cannot be enqueued. However there are different > scenarios when backoff could happen. Currently there is no way to > differenciate whether a backoff happened due to lowest prio+disconnection or > queue overflow from higher priority queues when connection between client and > namenode remains open. Currently IPC server just emits a single metrics for > all the backoffs. > Example: > # Client are directly enqueued into lowest priority queue and backoff when > lowest queue is full. Client are expected to disconnect from namenode. > # Client are enqueued into non-lowest priority queue and overflowed all the > way down to lowest priority queue and back off. In this case, connection > between client and namenode remains open. > We would like to add metrics for #1 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16874) Improve DataNode decommission for Erasure Coding
[ https://issues.apache.org/jira/browse/HDFS-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804427#comment-17804427 ] Bryan Beaudreault commented on HDFS-16874: -- [~jingzhao] is there any update here? > Improve DataNode decommission for Erasure Coding > > > Key: HDFS-16874 > URL: https://issues.apache.org/jira/browse/HDFS-16874 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ec, erasure-coding >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Major > > There are a couple of issues with the current DataNode decommission > implementation when large amounts of Erasure Coding data are involved in the > data re-replication/reconstruction process: > # Slowness. In HDFS-8786 we made a decision to use re-replication for > DataNode decommission if the internal EC block is still available. While this > strategy reduces the CPU cost caused by EC reconstruction, it greatly limits > the overall data recovery bandwidth, since there is only one single DataNode > as the source. While high density HDD hosts are more and more widely used by > HDFS especially along with Erasure Coding for warm data use case, this > becomes a big pain for cluster management. In our production, to decommission > a DataNode with several hundred TB EC data stored might take several days. > HDFS-16613 provides optimization based on the existing mechanism, but more > fundamentally we may want to allow EC reconstruction for DataNode > decommission so as to achieve much larger recovery bandwidth. > # The semantic of the existing EC reconstruction command (the > BlockECReconstructionInfoProto msg sent from NN to DN) is not clear. The > existing reconstruction command depends on the holes in the > srcNodes/liveBlockIndices arrays to indicate the target internal blocks for > recovery, while the holes can also be caused by the fact that the > corresponding datanode is too busy so it cannot be used as the reconstruction > source. This causes the later DataNode side reconstruction may not be > consistent with the original intention. E.g., if the index of the missing > block is 6, and the datanode storing block 0 is busy, the src nodes in the > reconstruction command only cover blocks [1, 2, 3, 4, 5, 7, 8]. The target > datanode may reconstruct the internal block 0 instead of 6. HDFS-16566 is > working on this issue by indicating an excluding index list. More > fundamentally we can follow the same path but go a step further by adding an > optional field explicitly indicating the target block indices in the command > protobuf msg. With the extension the DataNode will no longer use the holes in > the src node array to "guess" the reconstruction targets. > Internally we have developed and applied fixes by following the above > directions. We have seen significant improvement (100+ times speed up) in > terms of datanode decommission speed for EC data. The more clear semantic of > the reconstruction command protobuf msg also help prevent potential data > corruption during the EC reconstruction. > We will use this ticket to track the similar fixes for the Apache releases. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15815) if required storageType are unavailable, log the failed reason during choosing Datanode
[
https://issues.apache.org/jira/browse/HDFS-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804354#comment-17804354
]
ASF GitHub Bot commented on HDFS-15815:
---
tomasbanet commented on PR #2882:
URL: https://github.com/apache/hadoop/pull/2882#issuecomment-1881386023
Just upgraded our cluster from 3.1.x to 3.3.x, and seeing this log line
printed hundreds of times per minute:
```
INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not
enough replicas was chosen. Reason:{NO_REQUIRED_STORAGE_TYPE=1}
```
We have a single rack topology, with replication factor of 3.
For now working around this issue by setting log level of
BlockPlacementPolicy to WARN in log4j.properties:
```
log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy=WARN
```
> if required storageType are unavailable, log the failed reason during
> choosing Datanode
>
>
> Key: HDFS-15815
> URL: https://issues.apache.org/jira/browse/HDFS-15815
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
>Reporter: Yang Yun
>Assignee: Yang Yun
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.1, 3.4.0, 3.2.3
>
> Attachments: HDFS-15815.001.patch, HDFS-15815.002.patch,
> HDFS-15815.003.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> For better debug, if required storageType are unavailable, log the failed
> reason "NO_REQUIRED_STORAGE_TYPE" when choosing Datanode.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17302) RBF: ProportionRouterRpcFairnessPolicyController-Sharing and isolation.
[
https://issues.apache.org/jira/browse/HDFS-17302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17804193#comment-17804193
]
ASF GitHub Bot commented on HDFS-17302:
---
hadoop-yetus commented on PR #6380:
URL: https://github.com/apache/hadoop/pull/6380#issuecomment-1880622573
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 33s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 38m 53s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 23m 9s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 162m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6380/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6380 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 0e6f2dd6d007 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 93d86ccf8c20b3d7bf2c261a58e5099e2be4fdba |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6380/5/testReport/ |
| Max. process+thread count | 2455 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6380/5/console |
|
