[jira] [Commented] (HDFS-17149) getBlockLocations RPC should use actual client ip to compute network distance when using RBF.
[ https://issues.apache.org/jira/browse/HDFS-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753895#comment-17753895 ] Xiaoqiao He commented on HDFS-17149: [~zhanghaobo], Thanks, IIRC, there are many and many times we have discussed about locality for RBF, and from now on, still not agreement for some cases, such as #getBlockLocations here. I think we should push it forwards again thus enhance RBF for prod cluster. cc [~elgoiri], [~ayushtkn] what do you think about? > getBlockLocations RPC should use actual client ip to compute network distance > when using RBF. > - > > Key: HDFS-17149 > URL: https://issues.apache.org/jira/browse/HDFS-17149 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > > Please correct me if i understand wrongly. Thanks. > Currently, when a getBlockLocations RPC forwards to namenode via router. > NameNode will use router ip address as client machine to compute network > distance against block's locations. See FSNamesystem#sortLocatedBlocks method > for more detailed information. > I think this compute method is not correct and should use actual client ip. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working
[ https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753894#comment-17753894 ] ASF GitHub Bot commented on HDFS-17138: --- Hexiaoqiao commented on code in PR #5921: URL: https://github.com/apache/hadoop/pull/5921#discussion_r1292958875 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java: ## @@ -81,7 +81,12 @@ class AbstractDelegationTokenSecretManagerhttps://github.com/apache/hadoop/blob/8d95c588d2df0048b0d3eb711d74bf34bf4ae3c4/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java#L428-L430 > RBF: We changed the hadoop.security.auth_to_local configuration of one > router, the other routers stopped working > > > Key: HDFS-17138 > URL: https://issues.apache.org/jira/browse/HDFS-17138 > Project: Hadoop HDFS > Issue Type: Bug > Environment: hadoop 3.3.0 >Reporter: Xiping Zhang >Assignee: Xiping Zhang >Priority: Major > Labels: pull-request-available > Attachments: image-2023-08-02-16-20-34-454.png, > image-2023-08-03-10-32-03-457.png > > > other routers error log： > !image-2023-08-02-16-20-34-454.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17150) EC: Fix the bug of failed lease recovery.
[
https://issues.apache.org/jira/browse/HDFS-17150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753889#comment-17753889
]
ASF GitHub Bot commented on HDFS-17150:
---
Hexiaoqiao commented on code in PR #5937:
URL: https://github.com/apache/hadoop/pull/5937#discussion_r1292951771
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3802,16 +3803,26 @@ boolean internalReleaseLease(Lease lease, String src,
INodesInPath iip,
lastBlock.getBlockType());
}
- if (uc.getNumExpectedLocations() == 0 && lastBlock.getNumBytes() == 0) {
+ int minLocationsNum = 1;
+ if (lastBlock.isStriped()) {
+minLocationsNum = ((BlockInfoStriped) lastBlock).getRealDataBlockNum();
+ }
+ if (uc.getNumExpectedLocations() < minLocationsNum &&
+ lastBlock.getNumBytes() == 0) {
// There is no datanode reported to this block.
// may be client have crashed before writing data to pipeline.
// This blocks doesn't need any recovery.
// We can remove this block and close the file.
pendingFile.removeLastBlock(lastBlock);
finalizeINodeFileUnderConstruction(src, pendingFile,
iip.getLatestSnapshotId(), false);
-NameNode.stateChangeLog.warn("BLOCK* internalReleaseLease: "
-+ "Removed empty last block and closed file " + src);
+if (uc.getNumExpectedLocations() == 0) {
+ NameNode.stateChangeLog.warn("BLOCK* internalReleaseLease: "
+ + "Removed empty last block and closed file " + src);
+} else {
+ NameNode.stateChangeLog.warn("BLOCK* internalReleaseLease: "
Review Comment:
Totally true, but poor readability. Any other way to improve it, such as
`lastBlock.isStriped()` or others?
> EC: Fix the bug of failed lease recovery.
> -
>
> Key: HDFS-17150
> URL: https://issues.apache.org/jira/browse/HDFS-17150
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shuyan Zhang
>Priority: Major
> Labels: pull-request-available
>
> If the client crashes without writing the minimum number of internal blocks
> required by the EC policy, the lease recovery process for the corresponding
> unclosed file may continue to fail. Taking RS(6,3) policy as an example, the
> timeline is as follows:
> 1. The client writes some data to only 5 datanodes;
> 2. Client crashes;
> 3. NN fails over;
> 4. Now the result of `uc.getNumExpectedLocations()` completely depends on
> block report, and there are 5 datanodes reporting internal blocks;
> 5. When the lease expires hard limit, NN issues a block recovery command;
> 6. The datanode checks the command and finds that the number of internal
> blocks is insufficient, resulting in an error and recovery failure;
> 7. The lease expires hard limit again, and NN issues a block recovery command
> again, but the recovery fails again..
> When the number of internal blocks written by the client is less than 6, the
> block group is actually unrecoverable. We should equate this situation to the
> case where the number of replicas is 0 when processing replica files, i.e.,
> directly remove the last block group and close the file.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17156) mapreduce job encounters java.io.IOException when dfs.client.rbf.observer.read.enable is true
[
https://issues.apache.org/jira/browse/HDFS-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753871#comment-17753871
]
Chunyi Yang commented on HDFS-17156:
[~tasanuma] Thank you! I would like to work on this, but couldn't find the
'Assign to Me' option. Could you please advise me where to apply for the
assignee permission? Thanks in advance.
> mapreduce job encounters java.io.IOException when
> dfs.client.rbf.observer.read.enable is true
> -
>
> Key: HDFS-17156
> URL: https://issues.apache.org/jira/browse/HDFS-17156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Chunyi Yang
>Priority: Minor
> Labels: Observer, RBF
>
> While executing a mapreduce job in an environment utilizing Router-Based
> Federation with Observer read enabled, there is an estimated 1% chance of
> encountering the following error.
> {code:java}
> "java.io.IOException: Resource
> hdfs:///user//.staging/job_XX/.tez/application_XX/tez-conf.pb
> changed on src filesystem - expected: \"2023-07-07T12:41:16.801+0900\", was:
> \"2023-07-07T12:41:16.822+0900\", current time:
> \"2023-07-07T12:41:22.386+0900\"",
> {code}
> This error happens in function verifyAndCopy inside FSDownload.java when
> nodemanager tries to download a file right after the file has been written to
> the HDFS. The write operation runs on active namenode and read operation runs
> on observer namenode as expected.
> The edits file and hdfs-audit files indicate that the expected timestamp
> mentioned in the error message aligns with the OP_CLOSE MTIME of the
> 'tez-conf.pb' file (which is correct). However, the actual timestamp
> retrieved from the read operation corresponds to the OP_ADD MTIME of the
> target 'tez-conf.pf' file (which is incorrect). This inconsistency suggests
> that the observer namenode responds to the client before its edits file is
> updated with the latest stateId.
> Further troubleshooting has revealed that during write operations, the router
> responds to the client before receiving the latest stateId from the active
> namenode. Consequently, the outdated stateId is then used in the subsequent
> read operation on the observer namenode, leading to inaccuracies in the
> information provided by the observer namenode.
> To resolve this issue, it is essential to ensure that the router sends a
> response to the client only after receiving the latest stateId from the
> active namenode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working
[ https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753866#comment-17753866 ] ASF GitHub Bot commented on HDFS-17138: --- zhangxiping1 commented on code in PR #5921: URL: https://github.com/apache/hadoop/pull/5921#discussion_r1292904295 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java: ## @@ -81,7 +81,12 @@ class AbstractDelegationTokenSecretManager RBF: We changed the hadoop.security.auth_to_local configuration of one > router, the other routers stopped working > > > Key: HDFS-17138 > URL: https://issues.apache.org/jira/browse/HDFS-17138 > Project: Hadoop HDFS > Issue Type: Bug > Environment: hadoop 3.3.0 >Reporter: Xiping Zhang >Assignee: Xiping Zhang >Priority: Major > Labels: pull-request-available > Attachments: image-2023-08-02-16-20-34-454.png, > image-2023-08-03-10-32-03-457.png > > > other routers error log： > !image-2023-08-02-16-20-34-454.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working
[
https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753865#comment-17753865
]
ASF GitHub Bot commented on HDFS-17138:
---
zhangxiping1 commented on code in PR #5921:
URL: https://github.com/apache/hadoop/pull/5921#discussion_r1292903312
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java:
##
@@ -376,4 +382,61 @@ public void testDelegationTokenIdentifierToString() throws
Exception {
" for SomeUser with renewer JobTracker",
dtId.toStringStable());
}
+
+ public static class MyDelegationTokenSecretManager extends
+ AbstractDelegationTokenSecretManager {
+/**
+ * Create a secret manager
+ *
+ * @param delegationKeyUpdateIntervalthe number of milliseconds
for rolling
+ * new secret keys.
+ * @param delegationTokenMaxLifetime the maximum lifetime of the
delegation
+ * tokens in milliseconds
+ * @param delegationTokenRenewInterval how often the tokens must be
renewed
+ * in milliseconds
+ * @param delegationTokenRemoverScanInterval how often the tokens are
scanned
+ * for expired tokens in
milliseconds
+ */
+public MyDelegationTokenSecretManager(long delegationKeyUpdateInterval,
+long delegationTokenMaxLifetime, long delegationTokenRenewInterval,
+long delegationTokenRemoverScanInterval) {
+ super(delegationKeyUpdateInterval,
+ delegationTokenMaxLifetime,
+ delegationTokenRenewInterval,
+ delegationTokenRemoverScanInterval);
+}
+
+@Override
+public DelegationTokenIdentifier createIdentifier() {
+ return null;
+}
+
+@Override
+public void logExpireTokens(Collection
expiredTokens) throws IOException {
+ super.logExpireTokens(expiredTokens);
+}
+ }
+
+ @Test
+ public void testLogExpireTokensWhenChangeRules() {
+MyDelegationTokenSecretManager myDtSecretManager =
+new MyDelegationTokenSecretManager(10 * 1000, 10 * 1000, 10 * 1000, 10
* 1000);
+setRules("RULE:[2:$1@$0](SomeUser.*)s/.*/SomeUser/");
+DelegationTokenIdentifier dtId = new DelegationTokenIdentifier(
+new Text("SomeUser/h...@example.com"),
+new Text("SomeUser/h...@example.com"),
+new Text("SomeUser/h...@example.com"));
+Set expiredTokens = new HashSet();
+expiredTokens.add(dtId);
+
+setRules("RULE:[2:$1@$0](OtherUser.*)s/.*/OtherUser/");
+// rules was modified, causing the existing tokens (May be loaded from
other storage systems like zookeeper)
+// to fail to match the kerberos rules,
+// return an exception that cannot be handled
+try {
+ myDtSecretManager.logExpireTokens(expiredTokens);
+} catch (Exception e) {
+ Assert.fail("Exception in logExpireTokens");
Review Comment:
After fixing the code, there will be no exceptions, and if there are
exceptions, the code has not been fixed properly.
> RBF: We changed the hadoop.security.auth_to_local configuration of one
> router, the other routers stopped working
>
>
> Key: HDFS-17138
> URL: https://issues.apache.org/jira/browse/HDFS-17138
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: hadoop 3.3.0
>Reporter: Xiping Zhang
>Assignee: Xiping Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-08-02-16-20-34-454.png,
> image-2023-08-03-10-32-03-457.png
>
>
> other routers error log：
> !image-2023-08-02-16-20-34-454.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17138) RBF: We changed the hadoop.security.auth_to_local configuration of one router, the other routers stopped working
[
https://issues.apache.org/jira/browse/HDFS-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753864#comment-17753864
]
ASF GitHub Bot commented on HDFS-17138:
---
zhangxiping1 commented on code in PR #5921:
URL: https://github.com/apache/hadoop/pull/5921#discussion_r1292902814
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java:
##
@@ -376,4 +382,61 @@ public void testDelegationTokenIdentifierToString() throws
Exception {
" for SomeUser with renewer JobTracker",
dtId.toStringStable());
}
+
+ public static class MyDelegationTokenSecretManager extends
+ AbstractDelegationTokenSecretManager {
+/**
+ * Create a secret manager
+ *
+ * @param delegationKeyUpdateIntervalthe number of milliseconds
for rolling
+ * new secret keys.
+ * @param delegationTokenMaxLifetime the maximum lifetime of the
delegation
+ * tokens in milliseconds
+ * @param delegationTokenRenewInterval how often the tokens must be
renewed
+ * in milliseconds
+ * @param delegationTokenRemoverScanInterval how often the tokens are
scanned
+ * for expired tokens in
milliseconds
+ */
+public MyDelegationTokenSecretManager(long delegationKeyUpdateInterval,
+long delegationTokenMaxLifetime, long delegationTokenRenewInterval,
+long delegationTokenRemoverScanInterval) {
+ super(delegationKeyUpdateInterval,
Review Comment:
Yes, the ExpiredTokenRemover thread will operate automatically, which is
selected to be called automatically in the test case to determine if an
exception has been generated. If an exception has been generated, it indicates
that it has not been fixed.Any suggestions for improvement?
> RBF: We changed the hadoop.security.auth_to_local configuration of one
> router, the other routers stopped working
>
>
> Key: HDFS-17138
> URL: https://issues.apache.org/jira/browse/HDFS-17138
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: hadoop 3.3.0
>Reporter: Xiping Zhang
>Assignee: Xiping Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-08-02-16-20-34-454.png,
> image-2023-08-03-10-32-03-457.png
>
>
> other routers error log：
> !image-2023-08-02-16-20-34-454.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17157) Transient network failure in lease recovery could be mitigated to ensure better consistency
[
https://issues.apache.org/jira/browse/HDFS-17157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haoze Wu updated HDFS-17157:
Affects Version/s: 3.3.6
> Transient network failure in lease recovery could be mitigated to ensure
> better consistency
> ---
>
> Key: HDFS-17157
> URL: https://issues.apache.org/jira/browse/HDFS-17157
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.0.0-alpha, 3.3.6
>Reporter: Haoze Wu
>Priority: Major
>
> This case is related to HDFS-12070.
> In HDFS-12070, we saw how a faulty drive at a certain datanode could lead to
> permanent block recovery failure and leaves the file open indefinitely. In
> the patch, instead of failing the whole lease recovery process when the
> second stage of block recovery is failed at one datanode, the whole lease
> recovery process is failed if only these are failed for all the datanodes.
> Attached is the code snippet for the second stage of the block recovery, in
> BlockRecoveryWorker#syncBlock:
> {code:java}
> ...
> final List successList = new ArrayList<>();
> for (BlockRecord r : participatingList) {
> try {
> r.updateReplicaUnderRecovery(bpid, recoveryId, blockId,
> newBlock.getNumBytes());
> successList.add(r);
> } catch (IOException e) {
> ...{code}
> However, because of transient network failure, the RPC in
> updateReplicaUnderRecovery initiated from the primary datanode to another
> datanode could return an EOFException while the other side does not process
> the RPC at all or throw an IOException when reading from the socket.
> {code:java}
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:788)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy29.updateReplicaUnderRecovery(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolTranslatorPB.updateReplicaUnderRecovery(InterDatanodeProtocolTranslatorPB.java:112)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.updateReplicaUnderRecovery(BlockRecoveryWorker.java:88)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.access$700(BlockRecoveryWorker.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.syncBlock(BlockRecoveryWorker.java:300)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:188)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:606)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1796)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1165)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1061)
> {code}
> Then if there is any other datanode in which the second stage of block
> recovery success, the lease recovery would be successful and close the file.
> However, the last block failed to be synced to that failed datanode and this
> inconsistency could potentially last for a very long time.
> To fix the issue, I propose adding a configurable retry of
> updateReplicaUnderRecovery RPC so that transient network failure could be
> mitigated.
> Any comments and suggestions would be appreciated.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
