[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823112#comment-17823112 ] ASF GitHub Bot commented on HDFS-17387: --- ferhui merged PR #6572: URL: https://github.com/apache/hadoop/pull/6572 > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823113#comment-17823113 ] ASF GitHub Bot commented on HDFS-17387: --- ferhui commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1976195600 Thanks. Merged > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823094#comment-17823094 ] ASF GitHub Bot commented on HDFS-17387: --- ZanderXu commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1976061312 @ferhui The failed UTs are not introduced by this PR. I will submit a new PR to fix them and contribute them to the trunk branch. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[
https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823084#comment-17823084
]
ASF GitHub Bot commented on HDFS-17387:
---
hadoop-yetus commented on PR #6572:
URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1976013604
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 52s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 0m 46s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 39s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 40s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 0m 45s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 0m 42s | | HDFS-17384 passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 3s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 46s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 20m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 310 unchanged - 10
fixed = 310 total (was 320) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 201m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 298m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6572 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 084b55dedc03 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 / 50207fa1b5c523991566fbcf9d37169d9ee3ecf8 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
|
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823025#comment-17823025 ] ASF GitHub Bot commented on HDFS-17387: --- ferhui commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1975637977 seems the checkstyle issue still exists. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[
https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822547#comment-17822547
]
ASF GitHub Bot commented on HDFS-17387:
---
hadoop-yetus commented on PR #6572:
URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1973147848
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 16s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 0m 43s | | HDFS-17384 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 40s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 40s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 0m 40s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 0m 42s | | HDFS-17384 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 5s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 41s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 24m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 34s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/9/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 312 unchanged
- 8 fixed = 313 total (was 320) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 200m 13s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 292m 37s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6572 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 2fa4b80431c5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 /
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[
https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822424#comment-17822424
]
ASF GitHub Bot commented on HDFS-17387:
---
hadoop-yetus commented on PR #6572:
URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1972669698
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ HDFS-17384 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 49s | | HDFS-17384 passed |
| +1 :green_heart: | compile | 0m 44s | | HDFS-17384 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 40s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 41s | | HDFS-17384 passed |
| +1 :green_heart: | mvnsite | 0m 46s | | HDFS-17384 passed |
| +1 :green_heart: | javadoc | 0m 43s | | HDFS-17384 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | HDFS-17384 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 43s | | HDFS-17384 passed |
| +1 :green_heart: | shadedclient | 20m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 30s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/8/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 47 new + 312 unchanged
- 8 fixed = 359 total (was 320) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 205m 58s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 294m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6572/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6572 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 8969e8042f4e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17384 /
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822340#comment-17822340 ] ASF GitHub Bot commented on HDFS-17387: --- ZanderXu commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1972341796 > @ZanderXu @ferhui Thank you for work! I tried to upgrade the maven-surefire-plugin (#6537) on the trunk, but it was unsuccessful, which may cause some unit tests to fail to run. I rolled back this pr (#6578), and I cherrypicked this pr(#6578) to the [HDFS-17384](https://issues.apache.org/jira/browse/HDFS-17384) branch. @slfan1989 Thanks so much. I will rebase this PR again. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822301#comment-17822301 ] ASF GitHub Bot commented on HDFS-17387: --- slfan1989 commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1971955704 @ZanderXu @ferhui Thank you for work! I tried to upgrade the maven-surefire-plugin (#6537) on the trunk, but it was unsuccessful, which may cause some unit tests to fail to run. I rolled back this pr (#6578), and I cherrypicked this pr(#6578) to the HDFS-17384 branch. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[
https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821487#comment-17821487
]
ASF GitHub Bot commented on HDFS-17387:
---
ZanderXu commented on code in PR #6572:
URL: https://github.com/apache/hadoop/pull/6572#discussion_r1505263247
##
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##
@@ -3956,6 +3956,15 @@
+
+ dfs.namenode.lock.model.provider.class
+
org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock
+
+An implementation class of FSNamesystem lock.
+Defaults to GlobalFSNamesystemLock.class
Review Comment:
done
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -871,7 +876,10 @@ static FSNamesystem loadFromDisk(Configuration conf)
throws IOException {
this.contextFieldSeparator =
conf.get(HADOOP_CALLER_CONTEXT_SEPARATOR_KEY,
HADOOP_CALLER_CONTEXT_SEPARATOR_DEFAULT);
-fsLock = new FSNamesystemLock(conf, detailedLockHoldTimeMetrics);
+Class lockKlass = conf.getClass(
+DFS_NAMENODE_LOCK_MODEL_PROVIDER_KEY,
DFS_NAMENODE_LOCK_MODEL_PROVIDER_DEFAULT,
+FSNamesystemLock.class);
+fsLock = createLock(lockKlass, conf, detailedLockHoldTimeMetrics);
Review Comment:
done
> [FGL] Abstract the configurable locking mode
>
>
> Key: HDFS-17387
> URL: https://issues.apache.org/jira/browse/HDFS-17387
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> Abstract a lock mode to cover the current global lock and the new
> fine-grained lock(global FS lock and global BM lock).
> End-user can select to use lock mode through configuration.
> The possible lock modes after this patch are as follows:
> * GLOBAL Lock
> * FS Lock
> * BM Lock
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821486#comment-17821486 ] ASF GitHub Bot commented on HDFS-17387: --- ZanderXu commented on code in PR #6572: URL: https://github.com/apache/hadoop/pull/6572#discussion_r1505263154 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/MeasurableReentrantReadWriteLock.java: ## Review Comment: done > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821146#comment-17821146 ] ASF GitHub Bot commented on HDFS-17387: --- ferhui commented on code in PR #6572: URL: https://github.com/apache/hadoop/pull/6572#discussion_r1503967442 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/MeasurableReentrantReadWriteLock.java: ## Review Comment: Got it. If so. seems we can also add it into another ticket as an improvement > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821143#comment-17821143 ] ASF GitHub Bot commented on HDFS-17387: --- ZanderXu commented on code in PR #6572: URL: https://github.com/apache/hadoop/pull/6572#discussion_r1503958822 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/MeasurableReentrantReadWriteLock.java: ## Review Comment: Yes, this class is used to tracking performance for locking. This class is cloned from the original FSNamesystemLock.class, and just modifies some logic related to lock name and some comments. So that the GlobalLock and FGL can use this class to trace performance with different lock name. GlobalFSNamesystemLock is using this class to trace performance right now. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821129#comment-17821129 ] ASF GitHub Bot commented on HDFS-17387: --- ferhui commented on code in PR #6572: URL: https://github.com/apache/hadoop/pull/6572#discussion_r1503944791 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/MeasurableReentrantReadWriteLock.java: ## Review Comment: Is it used for tracking performance? seems unrelated to make the locking mode configurable. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821002#comment-17821002 ] ASF GitHub Bot commented on HDFS-17387: --- ZanderXu commented on code in PR #6572: URL: https://github.com/apache/hadoop/pull/6572#discussion_r1503727005 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/FineGrainedFSNamesystemLock.java: ## Review Comment: Of course, I will create a new ticket only for this FGL implementation and describe it in detail. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[
https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821000#comment-17821000
]
ASF GitHub Bot commented on HDFS-17387:
---
ZanderXu commented on code in PR #6572:
URL: https://github.com/apache/hadoop/pull/6572#discussion_r1503725124
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -871,7 +876,10 @@ static FSNamesystem loadFromDisk(Configuration conf)
throws IOException {
this.contextFieldSeparator =
conf.get(HADOOP_CALLER_CONTEXT_SEPARATOR_KEY,
HADOOP_CALLER_CONTEXT_SEPARATOR_DEFAULT);
-fsLock = new FSNamesystemLock(conf, detailedLockHoldTimeMetrics);
+Class lockKlass = conf.getClass(
+DFS_NAMENODE_LOCK_MODEL_PROVIDER_KEY,
DFS_NAMENODE_LOCK_MODEL_PROVIDER_DEFAULT,
+FSNamesystemLock.class);
+fsLock = createLock(lockKlass, conf, detailedLockHoldTimeMetrics);
Review Comment:
Yes, `GlobalFSNamesystemLock` should be the default class.
But in order to find dead locks for the other PRs via UT, I just change this
value to FGL first.
After we complete all sub-tasks of this milestone, I will change the default
value to GlobalFSNamesystemLock.
> [FGL] Abstract the configurable locking mode
>
>
> Key: HDFS-17387
> URL: https://issues.apache.org/jira/browse/HDFS-17387
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> Abstract a lock mode to cover the current global lock and the new
> fine-grained lock(global FS lock and global BM lock).
> End-user can select to use lock mode through configuration.
> The possible lock modes after this patch are as follows:
> * GLOBAL Lock
> * FS Lock
> * BM Lock
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[
https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820992#comment-17820992
]
ASF GitHub Bot commented on HDFS-17387:
---
ferhui commented on code in PR #6572:
URL: https://github.com/apache/hadoop/pull/6572#discussion_r1503709489
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/FineGrainedFSNamesystemLock.java:
##
Review Comment:
How about introducing the FGL implementation in another ticket. it's will be
easy for review
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -871,7 +876,10 @@ static FSNamesystem loadFromDisk(Configuration conf)
throws IOException {
this.contextFieldSeparator =
conf.get(HADOOP_CALLER_CONTEXT_SEPARATOR_KEY,
HADOOP_CALLER_CONTEXT_SEPARATOR_DEFAULT);
-fsLock = new FSNamesystemLock(conf, detailedLockHoldTimeMetrics);
+Class lockKlass = conf.getClass(
+DFS_NAMENODE_LOCK_MODEL_PROVIDER_KEY,
DFS_NAMENODE_LOCK_MODEL_PROVIDER_DEFAULT,
+FSNamesystemLock.class);
+fsLock = createLock(lockKlass, conf, detailedLockHoldTimeMetrics);
Review Comment:
This is used for the FGL implementation, right? GlobalFSNamesystemLock is
the original lock? If Im right, GlobalFSNamesystemLock should be the default
class?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##
@@ -3956,6 +3956,15 @@
+
+ dfs.namenode.lock.model.provider.class
+
org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock
+
+An implementation class of FSNamesystem lock.
+Defaults to GlobalFSNamesystemLock.class
Review Comment:
And here we should set GlobalFSNamesystemLock by default.
> [FGL] Abstract the configurable locking mode
>
>
> Key: HDFS-17387
> URL: https://issues.apache.org/jira/browse/HDFS-17387
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> Abstract a lock mode to cover the current global lock and the new
> fine-grained lock(global FS lock and global BM lock).
> End-user can select to use lock mode through configuration.
> The possible lock modes after this patch are as follows:
> * GLOBAL Lock
> * FS Lock
> * BM Lock
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820980#comment-17820980 ] ASF GitHub Bot commented on HDFS-17387: --- ZanderXu commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1965866817 > Since [HDFS-17394](https://issues.apache.org/jira/browse/HDFS-17394) has been merged. can rebase this PR. done > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17387) [FGL] Abstract the configurable locking mode
[ https://issues.apache.org/jira/browse/HDFS-17387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820979#comment-17820979 ] ASF GitHub Bot commented on HDFS-17387: --- ferhui commented on PR #6572: URL: https://github.com/apache/hadoop/pull/6572#issuecomment-1965862096 Since HDFS-17394 has been merged. can rebase this PR. > [FGL] Abstract the configurable locking mode > > > Key: HDFS-17387 > URL: https://issues.apache.org/jira/browse/HDFS-17387 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > Abstract a lock mode to cover the current global lock and the new > fine-grained lock(global FS lock and global BM lock). > End-user can select to use lock mode through configuration. > The possible lock modes after this patch are as follows: > * GLOBAL Lock > * FS Lock > * BM Lock -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
