[jira] [Commented] (HDFS-17511) method storagespaceConsumedContiguous should use BlockInfo#getReplication to compute dsDelta
[ https://issues.apache.org/jira/browse/HDFS-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844542#comment-17844542 ] ASF GitHub Bot commented on HDFS-17511: --- hfutatzhanghb commented on PR #6799: URL: https://github.com/apache/hadoop/pull/6799#issuecomment-2099796557 @Hexiaoqiao @zhangshuyan0 @tomscut @haiyang1987 Sir, plz help me review this PR when you have free time~ Thanks a lot. > method storagespaceConsumedContiguous should use BlockInfo#getReplication to > compute dsDelta > > > Key: HDFS-17511 > URL: https://issues.apache.org/jira/browse/HDFS-17511 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > > As title says, we should use BlockInfo#getReplication to compute storage > space in method INodeFile#storagespaceConsumedContiguous. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17510) Change of Codec configuration does not work
[
https://issues.apache.org/jira/browse/HDFS-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844510#comment-17844510
]
ASF GitHub Bot commented on HDFS-17510:
---
hadoop-yetus commented on PR #6798:
URL: https://github.com/apache/hadoop/pull/6798#issuecomment-2099678537
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 91m 16s | | trunk passed |
| +1 :green_heart: | compile | 39m 15s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| -1 :x: | mvnsite | 4m 05s |
[/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6798/1/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in trunk failed. |
| +1 :green_heart: | javadoc | 4m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 140m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 01s | | the patch passed |
| +1 :green_heart: | compile | 36m 12s | | the patch passed |
| +1 :green_heart: | javac | 36m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 21s | | the patch passed |
| -1 :x: | mvnsite | 4m 05s |
[/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6798/1/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| +1 :green_heart: | javadoc | 4m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 150m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 13s | | The patch does not
generate ASF License warnings. |
| | | 476m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6798 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 3e6272437ff7 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 88a3f9441aa4aa3cbea41b01be90b4a95881cef2 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6798/1/testReport/
|
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6798/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Change of Codec configuration does not work
> ---
>
> Key: HDFS-17510
> URL: https://issues.apache.org/jira/browse/HDFS-17510
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: compress
>Reporter: Zhikai Hu
>Priority: Minor
> Labels: pull-request-available
>
> In one of my projects, I need to dynamically adjust compression level for
> different files.
> However, I found that in most cases the new compression level does not take
> effect as expected, the old compression level continues to be used.
> Here is the relevant code snippet:
> ZStandardCodec zStandardCodec = new ZStandardCodec();
> zStandardCodec.setConf(conf);
> conf.set("io.compression.codec.zstd.level", "5"); // level may change
> dynamically
> conf.set("io.compression.codec.zstd", zStandardCodec.getClass().getName());
> writer = SequenceFile.createWriter(conf,
> SequenceFile.Writer.file(sequenceFilePath),
>
[jira] [Commented] (HDFS-17513) Fix reconfigure slow peer enable for Namenode.
[
https://issues.apache.org/jira/browse/HDFS-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844503#comment-17844503
]
ASF GitHub Bot commented on HDFS-17513:
---
hadoop-yetus commented on PR #6802:
URL: https://github.com/apache/hadoop/pull/6802#issuecomment-2099651764
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 92m 22s | | trunk passed |
| +1 :green_heart: | compile | 6m 22s | | trunk passed |
| +1 :green_heart: | checkstyle | 5m 05s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 00s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 153m 08s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 13s | | the patch passed |
| +1 :green_heart: | compile | 3m 43s | | the patch passed |
| +1 :green_heart: | javac | 3m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 167m 00s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 41s | | The patch does not
generate ASF License warnings. |
| | | 441m 09s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6802 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 b213c3ff39ba 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 017d97f7037f1ab95c7b93db2090841000e71830 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6802/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6802/1/console
|
| versions | git=2.45.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix reconfigure slow peer enable for Namenode.
> --
>
> Key: HDFS-17513
> URL: https://issues.apache.org/jira/browse/HDFS-17513
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Zhaobo Huang
>Assignee: Zhaobo Huang
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17511) method storagespaceConsumedContiguous should use BlockInfo#getReplication to compute dsDelta
[
https://issues.apache.org/jira/browse/HDFS-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844502#comment-17844502
]
ASF GitHub Bot commented on HDFS-17511:
---
hadoop-yetus commented on PR #6799:
URL: https://github.com/apache/hadoop/pull/6799#issuecomment-2099650464
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 00s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 92m 03s | | trunk passed |
| +1 :green_heart: | compile | 6m 31s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 05s | | trunk passed |
| +1 :green_heart: | shadedclient | 152m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 6m 07s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 26s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 165m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 27s | | The patch does not
generate ASF License warnings. |
| | | 439m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6799 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 1e88a5f1d281 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 2745319eaafd9c4625a99e7d5d83cf74e7e3029d |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6799/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6799/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> method storagespaceConsumedContiguous should use BlockInfo#getReplication to
> compute dsDelta
>
>
> Key: HDFS-17511
> URL: https://issues.apache.org/jira/browse/HDFS-17511
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
>
> As title says, we should use BlockInfo#getReplication to compute storage
> space in method INodeFile#storagespaceConsumedContiguous.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17476) fix: False positive "Observer Node is too far behind" due to long overflow.
[ https://issues.apache.org/jira/browse/HDFS-17476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844501#comment-17844501 ] ASF GitHub Bot commented on HDFS-17476: --- KeeProMise commented on PR #6747: URL: https://github.com/apache/hadoop/pull/6747#issuecomment-2099645135 @Hexiaoqiao @ZanderXu @haiyang1987 hi, If you have time, please help to take a look at this pr, thanks. > fix: False positive "Observer Node is too far behind" due to long overflow. > --- > > Key: HDFS-17476 > URL: https://issues.apache.org/jira/browse/HDFS-17476 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jian Zhang >Assignee: Jian Zhang >Priority: Critical > Labels: pull-request-available > Attachments: HDFS-17476.patch, image-2024-04-18-10-57-10-481.png > > > In the code GlobalStateIdContext#receiveRequestState(), if clientStateId is a > small negative number, clientStateId-serverStateId may be greater than > （ESTIMATED_TRANSACTIONS_PER_SECOND due to overflow > * TimeUnit.MILLISECONDS.toSeconds(clientWaitTime) > * ESTIMATED_SERVER_TIME_MULTIPLIER）, > resulting in false positives that Observer Node is too far behind. > !image-2024-04-18-10-57-10-481.png|width=742,height=110! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-2139) Fast copy for HDFS.
[ https://issues.apache.org/jira/browse/HDFS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844497#comment-17844497 ] liuguanghua commented on HDFS-2139: --- [~xuzq_zander] Hello,sir. The design doc can not be viewd because of permission. [https://docs.google.com/document/d/1OHdUpQmKD3TZ3xdmQsXNmlXJetn2QFPinMH31Q4BqkI/edit?usp=sharing] Can you upload a new version in Attachments? Thanks very much > Fast copy for HDFS. > --- > > Key: HDFS-2139 > URL: https://issues.apache.org/jira/browse/HDFS-2139 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Pritam Damania >Assignee: Rituraj >Priority: Major > Attachments: HDFS-2139-For-2.7.1.patch, HDFS-2139.patch, > HDFS-2139.patch, image-2022-08-11-11-48-17-994.png > > Original Estimate: 168h > Remaining Estimate: 168h > > There is a need to perform fast file copy on HDFS. The fast copy mechanism > for a file works as > follows : > 1) Query metadata for all blocks of the source file. > 2) For each block 'b' of the file, find out its datanode locations. > 3) For each block of the file, add an empty block to the namesystem for > the destination file. > 4) For each location of the block, instruct the datanode to make a local > copy of that block. > 5) Once each datanode has copied over its respective blocks, they > report to the namenode about it. > 6) Wait for all blocks to be copied and exit. > This would speed up the copying process considerably by removing top of > the rack data transfers. > Note : An extra improvement, would be to instruct the datanode to create a > hardlink of the block file if we are copying a block on the same datanode > [~xuzq_zander]Provided a design doc > https://docs.google.com/document/d/1OHdUpQmKD3TZ3xdmQsXNmlXJetn2QFPinMH31Q4BqkI/edit?usp=sharing -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17512) dumpXattrs logic optimization
[
https://issues.apache.org/jira/browse/HDFS-17512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844492#comment-17844492
]
ASF GitHub Bot commented on HDFS-17512:
---
hadoop-yetus commented on PR #6800:
URL: https://github.com/apache/hadoop/pull/6800#issuecomment-2099617359
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 01s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 00s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 00s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 01s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 00s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 87m 33s | | trunk passed |
| +1 :green_heart: | compile | 5m 45s | | trunk passed |
| +1 :green_heart: | checkstyle | 4m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 141m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 21s | | the patch passed |
| +1 :green_heart: | compile | 3m 16s | | the patch passed |
| +1 :green_heart: | javac | 3m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 01s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 151m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 5m 20s | | The patch does not
generate ASF License warnings. |
| | | 407m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6800 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | MINGW64_NT-10.0-17763 9f5ec32daf17 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 24abce356eb174aa9b97a7611a565e9697f0acc1 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6800/1/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6800/1/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> dumpXattrs logic optimization
> -
>
> Key: HDFS-17512
> URL: https://issues.apache.org/jira/browse/HDFS-17512
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0, 3.3.3
>Reporter: Yukun Zhang
>Priority: Minor
> Labels: pull-request-available
>
> The dumpXattrs logic in VIO should use
> FSImageFormatPBINode.Loader.loadXAttrs() to get the Xattrs attribute for easy
> maintenance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17513) Fix reconfigure slow peer enable for Namenode.
[ https://issues.apache.org/jira/browse/HDFS-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844483#comment-17844483 ] ASF GitHub Bot commented on HDFS-17513: --- huangzhaobo99 commented on PR #6802: URL: https://github.com/apache/hadoop/pull/6802#issuecomment-2099571733 @haiyang1987 Please take a look at this PR when you have time. If there are any configuration changes, NN may not be able to avoid slow nodes. Thanks. > Fix reconfigure slow peer enable for Namenode. > -- > > Key: HDFS-17513 > URL: https://issues.apache.org/jira/browse/HDFS-17513 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zhaobo Huang >Assignee: Zhaobo Huang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17366) NameNode Fine-Grained Locking via Namespace Tree
[ https://issues.apache.org/jira/browse/HDFS-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844480#comment-17844480 ] Shilun Fan commented on HDFS-17366: --- [~xuzq_zander] [~ferhui] I believe it is necessary to organize an online meeting for discussion in order to have a clearer understanding of the goals and development plans. Thank you once again! > NameNode Fine-Grained Locking via Namespace Tree > > > Key: HDFS-17366 > URL: https://issues.apache.org/jira/browse/HDFS-17366 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Attachments: NameNode Fine-Grained Locking Based On Directory Tree.pdf > > > As we all known, the write performance of NameNode is limited by the global > lock. We target to enable fine-grained locking based on the Namespace tree to > improve the performance of NameNode write operations. > There are multiple motivations for creating this ticket: > * We have implemented this fine-grained locking and gained nearly 7x > performance improvements in our prod environment > * Other companies made similar improvements based on their internal branch. > Internal branches are quite different from the community, so few feedback and > discussions in the community. > * The topic of fine-grained locking has been discussed for a very long time, > but still without any results. > > We implemented this fine-gained locking based on the namespace tree to > maximize the number of concurrency for disjoint or independent operations. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15634) Invalidate block on decommissioning DataNode after replication
[ https://issues.apache.org/jira/browse/HDFS-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844437#comment-17844437 ] Adam Binford edited comment on HDFS-15634 at 5/7/24 8:47 PM: - Chiming in on a 4 year old issue, we've hit similar issues where returning a node after decommissioning it cause the active namenode to failover because it holds the write lock too long processing the redundant blocks. In our case we have around ~1.2 million blocks on a single data node across three drives. I assume `DatanodeStorageInfo` represents a single drive? In which case the write lock is only released after processing all blocks on a single drive. was (Author: kimahriman): Chiming in on a 4 year old issue, we've hit similar issues where returning a node after decommissioning it cause the active namenode to failover because it holds the write lock too long processing the redundant blocks. In our case we have around ~120k blocks on a single data node across three drives. I assume `DatanodeStorageInfo` represents a single drive? In which case the write lock is only released after processing all blocks on a single drive. > Invalidate block on decommissioning DataNode after replication > -- > > Key: HDFS-15634 > URL: https://issues.apache.org/jira/browse/HDFS-15634 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Labels: pull-request-available > Attachments: write lock.png > > Time Spent: 1h > Remaining Estimate: 0h > > Right now when a DataNode starts decommission, Namenode will mark it as > decommissioning and its blocks will be replicated over to different > DataNodes, then marked as decommissioned. These blocks are not touched since > they are not counted as live replicas. > Proposal: Invalidate these blocks once they are replicated and there are > enough live replicas in the cluster. > Reason: A recent shutdown of decommissioned datanodes to finished the flow > caused Namenode latency spike since namenode needs to remove all of the > blocks from its memory and this step requires holding write lock. If we have > gradually invalidated these blocks the deletion will be much easier and > faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-17513) Fix reconfigure slow peer enable for Namenode.
[ https://issues.apache.org/jira/browse/HDFS-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaobo Huang reassigned HDFS-17513: --- Assignee: Zhaobo Huang > Fix reconfigure slow peer enable for Namenode. > -- > > Key: HDFS-17513 > URL: https://issues.apache.org/jira/browse/HDFS-17513 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zhaobo Huang >Assignee: Zhaobo Huang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17513) Fix reconfigure slow peer enable for Namenode.
[ https://issues.apache.org/jira/browse/HDFS-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17513: -- Labels: pull-request-available (was: ) > Fix reconfigure slow peer enable for Namenode. > -- > > Key: HDFS-17513 > URL: https://issues.apache.org/jira/browse/HDFS-17513 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zhaobo Huang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17513) Fix reconfigure slow peer enable for Namenode.
[ https://issues.apache.org/jira/browse/HDFS-17513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844225#comment-17844225 ] ASF GitHub Bot commented on HDFS-17513: --- huangzhaobo99 opened a new pull request, #6802: URL: https://github.com/apache/hadoop/pull/6802 ### Description of PR ### How was this patch tested? Add UT > Fix reconfigure slow peer enable for Namenode. > -- > > Key: HDFS-17513 > URL: https://issues.apache.org/jira/browse/HDFS-17513 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Zhaobo Huang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17513) Fix reconfigure slow peer enable for Namenode.
Zhaobo Huang created HDFS-17513: --- Summary: Fix reconfigure slow peer enable for Namenode. Key: HDFS-17513 URL: https://issues.apache.org/jira/browse/HDFS-17513 Project: Hadoop HDFS Issue Type: Bug Reporter: Zhaobo Huang -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17385) [FGL] Replace the global FS write locking with directory tree-based fine-grained locking
[ https://issues.apache.org/jira/browse/HDFS-17385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844164#comment-17844164 ] ZanderXu commented on HDFS-17385: - The branch HDFS-17385 is ready. > [FGL] Replace the global FS write locking with directory tree-based > fine-grained locking > > > Key: HDFS-17385 > URL: https://issues.apache.org/jira/browse/HDFS-17385 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL > > Second, we can replace the global FS write locking with directory tree-based > fine-grained locking. > This ticket doesn't remove this global FS locking, it just replaces the > global FS write lock with the global FS read lock and directory tree-based > fine-grained lock. > For block-related operations and DN-related operations still need the global > BM lock. > > The lock order should be: > * The global FS lock > * The global BM lock > * The directory tree-based fine-grained lock > > This ticket should supports: > * End-user can choose lock mode > ** One global lock mode > ** Global FS lock and global BM lock mode > ** Global FS read lock, global BM lock and tree-based fine-grained lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
