[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851033#comment-17851033 ] ASF GitHub Bot commented on HDFS-17384: --- zhengchenyu commented on PR #6762: URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2141744912 I know this PR is phase 1 work: split global lock into fslock and bmlock. We should ensure the lock order: bmlock cannot occur in front of fslock. Can we add some restrictions or check mechanisms to avoid getting a bmlock before fslock? > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL, pull-request-available > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851031#comment-17851031
]
ASF GitHub Bot commented on HDFS-17384:
---
zhengchenyu commented on code in PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#discussion_r1622166340
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/fgl/FineGrainedFSNamesystemLock.java:
##
@@ -0,0 +1,300 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.namenode.fgl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock;
+import org.apache.hadoop.metrics2.lib.MutableRatesWithAggregation;
+
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Supplier;
+
+/**
+ * Splitting the global FSN lock into FSLock and BMLock.
+ * FSLock is used to protect directory tree-related operations.
+ * BMLock is used to protect block-related and dn-related operations.
+ * The lock order should be: FSLock,BMLock.
+ */
+public class FineGrainedFSNamesystemLock implements FSNLockManager {
+ private final FSNamesystemLock fsLock;
+ private final FSNamesystemLock bmLock;
+
+ public FineGrainedFSNamesystemLock(Configuration conf,
MutableRatesWithAggregation aggregation) {
+this.fsLock = new FSNamesystemLock(conf, "FS", aggregation);
+this.bmLock = new FSNamesystemLock(conf, "BM", aggregation);
+ }
+
+ @Override
+ public void readLock(FSNamesystemLockMode lockMode) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.fsLock.readLock();
+ this.bmLock.readLock();
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readLock();
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readLock();
+}
+ }
+
+ public void readLockInterruptibly(FSNamesystemLockMode lockMode) throws
InterruptedException {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.fsLock.readLockInterruptibly();
+ try {
+this.bmLock.readLockInterruptibly();
+ } catch (InterruptedException e) {
+// The held FSLock should be released if the current thread is
interrupted
+// while acquiring the BMLock.
+this.fsLock.readUnlock("BMReadLockInterruptiblyFailed");
+throw e;
+ }
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readLockInterruptibly();
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readLockInterruptibly();
+}
+ }
+
+ @Override
+ public void readUnlock(FSNamesystemLockMode lockMode, String opName) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.bmLock.readUnlock(opName);
+ this.fsLock.readUnlock(opName);
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readUnlock(opName);
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readUnlock(opName);
+}
+ }
+
+ public void readUnlock(FSNamesystemLockMode lockMode, String opName,
+ Supplier lockReportInfoSupplier) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.bmLock.readUnlock(opName, lockReportInfoSupplier);
+ this.fsLock.readUnlock(opName, lockReportInfoSupplier);
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.readUnlock(opName, lockReportInfoSupplier);
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.readUnlock(opName, lockReportInfoSupplier);
+}
+ }
+
+ @Override
+ public void writeLock(FSNamesystemLockMode lockMode) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+ this.fsLock.writeLock();
+ this.bmLock.writeLock();
+} else if (lockMode.equals(FSNamesystemLockMode.FS)) {
+ this.fsLock.writeLock();
+} else if (lockMode.equals(FSNamesystemLockMode.BM)) {
+ this.bmLock.writeLock();
+}
+ }
+
+ @Override
+ public void writeUnlock(FSNamesystemLockMode lockMode, String opName) {
+if (lockMode.equals(FSNamesystemLockMode.GLOBAL)) {
+
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845776#comment-17845776 ] ASF GitHub Bot commented on HDFS-17384: --- hadoop-yetus commented on PR #6762: URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2106553639 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | -1 :x: | patch | 1m 02s | | https://github.com/apache/hadoop/pull/6762 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/6762 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6762/3/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL, pull-request-available > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844096#comment-17844096
]
ASF GitHub Bot commented on HDFS-17384:
---
ZanderXu commented on code in PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#discussion_r1591744778
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -6043,7 +6079,7 @@ void updatePipeline(
updatePipelineInternal(clientName, oldBlock, newBlock, newNodes,
newStorageIDs, logRetryCache);
} finally {
- writeUnlock("updatePipeline");
+ writeUnlock(FSNamesystemLockMode.GLOBAL, "updatePipeline");
Review Comment:
Yes, this RPC involves iNode and FSEditLog, so we use global lock here.
> [FGL] Replace the global lock with global FS Lock and global BM lock
>
>
> Key: HDFS-17384
> URL: https://issues.apache.org/jira/browse/HDFS-17384
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: FGL, pull-request-available
>
> First, we can replace the current global lock with two locks, global FS lock
> and global BM lock.
> The global FS lock is used to make directory tree-related operations
> thread-safe.
> The global BM lock is used to make block-related operations and DN-related
> operations thread-safe.
>
> For some operations involving both directory tree and block or DN, the global
> FS lock and the global BM lock are acquired.
>
> The lock order should be:
> * The global FS lock
> * The global BM lock
>
> There are some special requirements for this ticket.
> * End-user can choose to use global lock or fine-grained lock through
> configuration.
> * Try not to modify the current implementation logic as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843519#comment-17843519
]
ASF GitHub Bot commented on HDFS-17384:
---
ferhui commented on code in PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#discussion_r1590192211
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -6043,7 +6079,7 @@ void updatePipeline(
updatePipelineInternal(clientName, oldBlock, newBlock, newNodes,
newStorageIDs, logRetryCache);
} finally {
- writeUnlock("updatePipeline");
+ writeUnlock(FSNamesystemLockMode.GLOBAL, "updatePipeline");
Review Comment:
That documentation is the final solution. Currently this PR is the 1st
stage. This stage only aims to split the global lock into bm and fs locks.
In updatePipelineInternal we can see it uses inode too, so we use global
lock here. Will solve similar cases in 2nd stage.
@ZanderXu any other comments?
> [FGL] Replace the global lock with global FS Lock and global BM lock
>
>
> Key: HDFS-17384
> URL: https://issues.apache.org/jira/browse/HDFS-17384
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: FGL, pull-request-available
>
> First, we can replace the current global lock with two locks, global FS lock
> and global BM lock.
> The global FS lock is used to make directory tree-related operations
> thread-safe.
> The global BM lock is used to make block-related operations and DN-related
> operations thread-safe.
>
> For some operations involving both directory tree and block or DN, the global
> FS lock and the global BM lock are acquired.
>
> The lock order should be:
> * The global FS lock
> * The global BM lock
>
> There are some special requirements for this ticket.
> * End-user can choose to use global lock or fine-grained lock through
> configuration.
> * Try not to modify the current implementation logic as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841789#comment-17841789
]
ASF GitHub Bot commented on HDFS-17384:
---
hadoop-yetus commented on PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2081720138
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 14s | | No case conflicting files
found. |
| +0 :ok: | spotbugs | 0m 01s | | spotbugs executables are not
available. |
| +0 :ok: | codespell | 0m 01s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 01s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 01s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 00s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 00s | | The patch appears to
include 47 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 87m 56s | | trunk passed |
| +1 :green_heart: | compile | 39m 10s | | trunk passed |
| +1 :green_heart: | checkstyle | 7m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 15m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 14m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 174m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 2m 14s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 9m 51s | | the patch passed |
| +1 :green_heart: | compile | 36m 45s | | the patch passed |
| +1 :green_heart: | javac | 36m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 01s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 5m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 16m 04s | | the patch passed |
| +1 :green_heart: | javadoc | 14m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 183m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 6m 03s | | The patch does not
generate ASF License warnings. |
| | | 552m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/6762 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
markdownlint |
| uname | MINGW64_NT-10.0-17763 087b28dd158d 3.4.10-87d57229.x86_64
2024-02-14 20:17 UTC x86_64 Msys |
| Build tool | maven |
| Personality | /c/hadoop/dev-support/bin/hadoop.sh |
| git revision | trunk / 14153f07aab8a229f128f503b2741d1c81a824b2 |
| Default Java | Azul Systems, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6762/2/testReport/
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs-rbf hadoop-tools/hadoop-fs2img U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6762/2/console
|
| versions | git=2.44.0.windows.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> [FGL] Replace the global lock with global FS Lock and global BM lock
>
>
> Key: HDFS-17384
> URL: https://issues.apache.org/jira/browse/HDFS-17384
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: FGL, pull-request-available
>
> First, we can replace the current global lock with two locks, global FS lock
> and global BM lock.
> The global FS lock is used to make directory tree-related operations
> thread-safe.
> The global BM lock is used to make block-related operations and DN-related
> operations thread-safe.
>
> For some operations involving both directory tree and block or DN, the global
> FS lock and the global BM lock are acquired.
>
> The lock order should be:
> * The global FS lock
> * The global BM lock
>
> There are some special requirements for this ticket.
> * End-user can choose to use global lock or fine-grained lock through
>
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840628#comment-17840628
]
ASF GitHub Bot commented on HDFS-17384:
---
hfutatzhanghb commented on code in PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#discussion_r1578763344
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -6043,7 +6079,7 @@ void updatePipeline(
updatePipelineInternal(clientName, oldBlock, newBlock, newNodes,
newStorageIDs, logRetryCache);
} finally {
- writeUnlock("updatePipeline");
+ writeUnlock(FSNamesystemLockMode.GLOBAL, "updatePipeline");
Review Comment:
@ZanderXu Hi, sir. Have a question here. In documentaion, it is said that
updateBlockForPipeline and updatePipeline are only involved by blocks
operations. Why we use FSNamesystemLockMode.GLOBAL here?
> [FGL] Replace the global lock with global FS Lock and global BM lock
>
>
> Key: HDFS-17384
> URL: https://issues.apache.org/jira/browse/HDFS-17384
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: FGL, pull-request-available
>
> First, we can replace the current global lock with two locks, global FS lock
> and global BM lock.
> The global FS lock is used to make directory tree-related operations
> thread-safe.
> The global BM lock is used to make block-related operations and DN-related
> operations thread-safe.
>
> For some operations involving both directory tree and block or DN, the global
> FS lock and the global BM lock are acquired.
>
> The lock order should be:
> * The global FS lock
> * The global BM lock
>
> There are some special requirements for this ticket.
> * End-user can choose to use global lock or fine-grained lock through
> configuration.
> * Try not to modify the current implementation logic as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840057#comment-17840057 ] ASF GitHub Bot commented on HDFS-17384: --- ZanderXu commented on PR #6762: URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2071967284 The spotbugs will be fixed by HDFS-17451. And the failed UT is not imported by this PR. > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL, pull-request-available > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840054#comment-17840054
]
ASF GitHub Bot commented on HDFS-17384:
---
hadoop-yetus commented on PR #6762:
URL: https://github.com/apache/hadoop/pull/6762#issuecomment-2071959089
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 47 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 14s | | trunk passed |
| +1 :green_heart: | compile | 17m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 16m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 0s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 34s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 34m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | | the patch passed |
| +1 :green_heart: | compile | 16m 51s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 51s | | the patch passed |
| +1 :green_heart: | compile | 16m 21s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 16m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 36s | | root: The patch generated
0 new + 1232 unchanged - 16 fixed = 1232 total (was 1248) |
| +1 :green_heart: | mvnsite | 3m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 55s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 3m 32s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 6m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 237m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 29m 40s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | unit | 0m 50s | | hadoop-fs2img in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not
generate ASF License warnings. |
| | | 506m 7s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6762/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6762 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839915#comment-17839915 ] ASF GitHub Bot commented on HDFS-17384: --- ZanderXu opened a new pull request, #6762: URL: https://github.com/apache/hadoop/pull/6762 We plan to merge HDFS-17384 to the trunk branch. This PR is used to review all changes in HDFS-17384. > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[
https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835103#comment-17835103
]
ZanderXu commented on HDFS-17384:
-
{quote}[~xuzq_zander] Seems we also need a task to add document for this
feature, and then can merge it into master branch. What do you think of that?
{quote}
[~ferhui] Sure, I will create a ticket to do it.
> [FGL] Replace the global lock with global FS Lock and global BM lock
>
>
> Key: HDFS-17384
> URL: https://issues.apache.org/jira/browse/HDFS-17384
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: FGL
>
> First, we can replace the current global lock with two locks, global FS lock
> and global BM lock.
> The global FS lock is used to make directory tree-related operations
> thread-safe.
> The global BM lock is used to make block-related operations and DN-related
> operations thread-safe.
>
> For some operations involving both directory tree and block or DN, the global
> FS lock and the global BM lock are acquired.
>
> The lock order should be:
> * The global FS lock
> * The global BM lock
>
> There are some special requirements for this ticket.
> * End-user can choose to use global lock or fine-grained lock through
> configuration.
> * Try not to modify the current implementation logic as much as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17384) [FGL] Replace the global lock with global FS Lock and global BM lock
[ https://issues.apache.org/jira/browse/HDFS-17384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835093#comment-17835093 ] Hui Fei commented on HDFS-17384: [~xuzq_zander] Seems we also need a task to add document for this feature, and then can merge it into master branch. What do you think of that? > [FGL] Replace the global lock with global FS Lock and global BM lock > > > Key: HDFS-17384 > URL: https://issues.apache.org/jira/browse/HDFS-17384 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: FGL > > First, we can replace the current global lock with two locks, global FS lock > and global BM lock. > The global FS lock is used to make directory tree-related operations > thread-safe. > The global BM lock is used to make block-related operations and DN-related > operations thread-safe. > > For some operations involving both directory tree and block or DN, the global > FS lock and the global BM lock are acquired. > > The lock order should be: > * The global FS lock > * The global BM lock > > There are some special requirements for this ticket. > * End-user can choose to use global lock or fine-grained lock through > configuration. > * Try not to modify the current implementation logic as much as possible. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
