[jira] [Commented] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl
[ https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654806#comment-17654806 ] ASF GitHub Bot commented on HDFS-16848: --- ZanderXu commented on PR #5147: URL: https://github.com/apache/hadoop/pull/5147#issuecomment-1371846633 @howzi It seems the failed UT `hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver` is not caused by this PR, but can you fix it in a new PR? > RBF: Improve StateStoreZookeeperImpl > - > > Key: HDFS-16848 > URL: https://issues.apache.org/jira/browse/HDFS-16848 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Sun Hao >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Currently, router is getting/updating state from zk sequentially. It will > slowdown router load/update state cache especially for a large cluster or a > multi region cluster. > We propose adding a threadpool to deal with zk state synchronization。 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl
[
https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654802#comment-17654802
]
ASF GitHub Bot commented on HDFS-16848:
---
ZanderXu commented on code in PR #5147:
URL: https://github.com/apache/hadoop/pull/5147#discussion_r1062141662
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java:
##
@@ -109,8 +138,16 @@ public boolean initRecordStorage(
}
}
+ @VisibleForTesting
+ public void setEnableConcurrent(boolean enableConcurrent) {
+this.enableConcurrent = enableConcurrent;
+ }
+
@Override
public void close() throws Exception {
+if(executorService != null) {
Review Comment:
`if (executorService != null) {`
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java:
##
@@ -63,8 +72,14 @@ public class StateStoreZooKeeperImpl extends
StateStoreSerializableImpl {
RBFConfigKeys.FEDERATION_STORE_PREFIX + "driver.zk.";
public static final String FEDERATION_STORE_ZK_PARENT_PATH =
FEDERATION_STORE_ZK_DRIVER_PREFIX + "parent-path";
+ public static final String FEDERATION_STORE_ZK_CLIENT_THREADS_SIZE =
+ FEDERATION_STORE_ZK_DRIVER_PREFIX + "client.size";
Review Comment:
how about changing the name to `FEDERATION_STORE_ZK_DRIVER_PREFIX +
"async.max.threads"`?
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java:
##
@@ -63,8 +72,14 @@ public class StateStoreZooKeeperImpl extends
StateStoreSerializableImpl {
RBFConfigKeys.FEDERATION_STORE_PREFIX + "driver.zk.";
public static final String FEDERATION_STORE_ZK_PARENT_PATH =
FEDERATION_STORE_ZK_DRIVER_PREFIX + "parent-path";
+ public static final String FEDERATION_STORE_ZK_CLIENT_THREADS_SIZE =
+ FEDERATION_STORE_ZK_DRIVER_PREFIX + "client.size";
+ public static final int FEDERATION_STORE_ZK_CLIENT_THREADS_SIZE_DEFAULT = -1;
Review Comment:
This configuration should be moved to
`org.apache.hadoop.hdfs.server.federation.router.RBFConfigKeys` if you want to
add some descriptions in hdfs-rbf-default.xml
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java:
##
@@ -137,34 +174,22 @@ public QueryResult get(Class
clazz)
String znode = getZNodeForClass(clazz);
try {
List children = zkManager.getChildren(znode);
- for (String child : children) {
-try {
- String path = getNodePath(znode, child);
- Stat stat = new Stat();
- String data = zkManager.getStringData(path, stat);
- boolean corrupted = false;
- if (data == null || data.equals("")) {
-// All records should have data, otherwise this is corrupted
-corrupted = true;
- } else {
-try {
- T record = createRecord(data, stat, clazz);
- ret.add(record);
-} catch (IOException e) {
- LOG.error("Cannot create record type \"{}\" from \"{}\": {}",
- clazz.getSimpleName(), data, e.getMessage());
- corrupted = true;
-}
- }
-
- if (corrupted) {
-LOG.error("Cannot get data for {} at {}, cleaning corrupted data",
-child, path);
-zkManager.delete(path);
+ List> callables = new ArrayList<>();
+ if (enableConcurrent) {
+children.forEach(child -> callables.add(() -> getRecord(clazz, znode,
child)));
+List> futures = executorService.invokeAll(callables);
+for (Future future : futures) {
+ if (future.get() != null) {
+ret.add(future.get());
}
-} catch (Exception e) {
- LOG.error("Cannot get data for {}: {}", child, e.getMessage());
}
+ } else {
+children.forEach(child -> {
+ T record = getRecord(clazz, znode, child);
+ if (record != null) {
+ret.add(record);
+ }
+});
Review Comment:
```
List> callables = new ArrayList<>();
zkManager.getChildren(znode).forEach(c -> callables.add(() ->
getRecord(clazz, znode, c)));
if (enableConcurrent) {
List> futures = executorService.invokeAll(callables);
for (Future future : futures) {
if (future.get() != null) {
ret.add(future.get());
}
}
} else {
for (Callable callable : callables) {
T record = callable.call();
if (record != null) {
ret.add(record);
}
}
}
```
##
[jira] [Updated] (HDFS-16881) Warn if AccessControlEnforcer runs for a long time to check permission
[ https://issues.apache.org/jira/browse/HDFS-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-16881: - Fix Version/s: 3.4.0 (was: 1.3.0) > Warn if AccessControlEnforcer runs for a long time to check permission > -- > > Key: HDFS-16881 > URL: https://issues.apache.org/jira/browse/HDFS-16881 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > AccessControlEnforcer is configurable. If an external AccessControlEnforcer > runs for a long time to check permission with the FSnamesystem lock, it will > significantly slow down the entire Namenode. In the JIRA, we will print a > WARN message when it happens. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16865) RBF: The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC.
[
https://issues.apache.org/jira/browse/HDFS-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654719#comment-17654719
]
ASF GitHub Bot commented on HDFS-16865:
---
ZanderXu commented on code in PR #5200:
URL: https://github.com/apache/hadoop/pull/5200#discussion_r1062087729
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java:
##
@@ -979,7 +979,10 @@ public boolean complete(String src, String clientName,
ExtendedBlock last, long fileId)
throws IOException {
checkNNStartup();
-return namesystem.completeFile(src, clientName, last, fileId);
+boolean result = namesystem.completeFile(src, clientName, last, fileId);
+LOG.debug("complete: src={}, clientName={}, fileId={}, result={}.",
Review Comment:
@goiri Thanks sir, I have updated it, please help me review it again. Thanks
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -465,6 +465,24 @@ public void setOwner(String src, String username, String
groupname)
}
}
+ /**
+ * Try to get the remote location whose bpId is same with the input bpId
from the input locations.
+ * @param locations the input RemoteLocations.
+ * @param bpId the input bpId.
+ * @return the remote location whose bpId is same with the input.
+ * @throws IOException
+ */
+ private RemoteLocation getLocationWithBPID(List locations,
String bpId)
+ throws IOException {
+String nsId = rpcClient.getNameserviceForBlockPoolId(bpId);
+for (RemoteLocation l : locations) {
+ if (l.getNameserviceId().equals(nsId)) {
+return l;
+ }
+}
+throw new IOException("Can't found remote locations for the " + bpId);
Review Comment:
@ayushtkn Thanks sir, I have updated it, please help me review it again.
Thanks
> RBF: The source path is always / after RBF proxied the complete, addBlock and
> getAdditionalDatanode RPC.
>
>
> Key: HDFS-16865
> URL: https://issues.apache.org/jira/browse/HDFS-16865
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> The source path is always / after RBF proxied the complete, addBlock and
> getAdditionalDatanode RPC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[ https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-16884: -- Description: Since the default is async delete replica on the datanode, the replica may not be deleted during the execution of UT#testConcurrentWriteAndDeleteBlock, resulting in a mismatch between the number of replicas in each dataset obtained at the end and the expectation (was: Since the default is async delete replica on datanode, the replica may not be to complete deleted during the UT#testConcurrentWriteAndDeleteBlock execution process, resulting in the final result not meeting expectations) > Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed > -- > > Key: HDFS-16884 > URL: https://issues.apache.org/jira/browse/HDFS-16884 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > Since the default is async delete replica on the datanode, the replica may > not be deleted during the execution of UT#testConcurrentWriteAndDeleteBlock, > resulting in a mismatch between the number of replicas in each dataset > obtained at the end and the expectation -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[ https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-16884: -- Description: Since the default is async delete replica on datanode, the replica may not be to complete deleted during the UT#testConcurrentWriteAndDeleteBlock execution process, resulting in the final result not meeting expectations (was: Since the default is async delete replica on datanode, the replica may not be to complete deleted during the UT# execution process, resulting in the final result not meeting expectations) > Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed > -- > > Key: HDFS-16884 > URL: https://issues.apache.org/jira/browse/HDFS-16884 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > Since the default is async delete replica on datanode, the replica may not be > to complete deleted during the UT#testConcurrentWriteAndDeleteBlock execution > process, resulting in the final result not meeting expectations -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[ https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-16884: -- Description: Since the default is async delete replica on datanode, the replica may not be to complete deleted during the UT# execution process, resulting in the final result not meeting expectations (was: Since the default is async delete replica on datanode, the replica may not be to complete deleted during the UT execution process, resulting in the final result not meeting expectations) > Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed > -- > > Key: HDFS-16884 > URL: https://issues.apache.org/jira/browse/HDFS-16884 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > Since the default is async delete replica on datanode, the replica may not be > to complete deleted during the UT# execution process, resulting in the final > result not meeting expectations -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[ https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu updated HDFS-16884: -- Description: Since the default is async delete replica on datanode, the replica may not be to complete deleted during the UT execution process, resulting in the final result not meeting expectations > Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed > -- > > Key: HDFS-16884 > URL: https://issues.apache.org/jira/browse/HDFS-16884 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > > Since the default is async delete replica on datanode, the replica may not be > to complete deleted during the UT execution process, resulting in the final > result not meeting expectations -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
Haiyang Hu created HDFS-16884: - Summary: Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed Key: HDFS-16884 URL: https://issues.apache.org/jira/browse/HDFS-16884 Project: Hadoop HDFS Issue Type: Bug Reporter: Haiyang Hu -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16884) Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed
[ https://issues.apache.org/jira/browse/HDFS-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu reassigned HDFS-16884: - Assignee: Haiyang Hu > Fix TestFsDatasetImpl#testConcurrentWriteAndDeleteBlock failed > -- > > Key: HDFS-16884 > URL: https://issues.apache.org/jira/browse/HDFS-16884 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16767) RBF: Support observer node from Router-Based Federation
[ https://issues.apache.org/jira/browse/HDFS-16767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-16767: Fix Version/s: (was: 3.3.5) > RBF: Support observer node from Router-Based Federation > > > Key: HDFS-16767 > URL: https://issues.apache.org/jira/browse/HDFS-16767 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Enable routers to direct read calls to observer namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13522) HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-13522:
Fix Version/s: (was: 3.3.5)
> HDFS-13522: Add federated nameservices states to client protocol and
> propagate it between routers and clients.
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{{}FederationNamenodeServiceState{}}}.
> This patch captures the state of all namespaces in the routers and propagates
> it to clients. A follow up patch will change router behavior to direct
> requests to the observer.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13522) HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654706#comment-17654706
]
Takanobu Asanuma commented on HDFS-13522:
-
Thanks for your reply, [~simbadzina]. If I understand correctly, the fixed
versions should be updated when the version (branch) has included the feature.
So I'll remove 3.3.5 from the fixed versions for now. I hope branch-3.3 include
this great feature. Maybe I can help with it.
> HDFS-13522: Add federated nameservices states to client protocol and
> propagate it between routers and clients.
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{{}FederationNamenodeServiceState{}}}.
> This patch captures the state of all namespaces in the routers and propagates
> it to clients. A follow up patch will change router behavior to direct
> requests to the observer.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16881) Warn if AccessControlEnforcer runs for a long time to check permission
[ https://issues.apache.org/jira/browse/HDFS-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz-wo Sze resolved HDFS-16881. --- Fix Version/s: 1.3.0 Hadoop Flags: Reviewed Resolution: Fixed The pull request is now merged. > Warn if AccessControlEnforcer runs for a long time to check permission > -- > > Key: HDFS-16881 > URL: https://issues.apache.org/jira/browse/HDFS-16881 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Labels: pull-request-available > Fix For: 1.3.0 > > > AccessControlEnforcer is configurable. If an external AccessControlEnforcer > runs for a long time to check permission with the FSnamesystem lock, it will > significantly slow down the entire Namenode. In the JIRA, we will print a > WARN message when it happens. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16881) Warn if AccessControlEnforcer runs for a long time to check permission
[ https://issues.apache.org/jira/browse/HDFS-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654694#comment-17654694 ] ASF GitHub Bot commented on HDFS-16881: --- szetszwo commented on PR #5268: URL: https://github.com/apache/hadoop/pull/5268#issuecomment-1371632593 @cnauroth , thanks a lot for reviewing this! > Warn if AccessControlEnforcer runs for a long time to check permission > -- > > Key: HDFS-16881 > URL: https://issues.apache.org/jira/browse/HDFS-16881 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Labels: pull-request-available > > AccessControlEnforcer is configurable. If an external AccessControlEnforcer > runs for a long time to check permission with the FSnamesystem lock, it will > significantly slow down the entire Namenode. In the JIRA, we will print a > WARN message when it happens. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16881) Warn if AccessControlEnforcer runs for a long time to check permission
[ https://issues.apache.org/jira/browse/HDFS-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654695#comment-17654695 ] ASF GitHub Bot commented on HDFS-16881: --- szetszwo merged PR #5268: URL: https://github.com/apache/hadoop/pull/5268 > Warn if AccessControlEnforcer runs for a long time to check permission > -- > > Key: HDFS-16881 > URL: https://issues.apache.org/jira/browse/HDFS-16881 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Tsz-wo Sze >Assignee: Tsz-wo Sze >Priority: Major > Labels: pull-request-available > > AccessControlEnforcer is configurable. If an external AccessControlEnforcer > runs for a long time to check permission with the FSnamesystem lock, it will > significantly slow down the entire Namenode. In the JIRA, we will print a > WARN message when it happens. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13522) HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654691#comment-17654691
]
Simbarashe Dzinamarira edited comment on HDFS-13522 at 1/5/23 1:18 AM:
---
Hi [~tasanuma]. We haven't backported HDFS-13522 and HDFS-16767 to the 3.x line
yet. There were large merge conflicts so we deferred the work.
So far, only HDFS-16669 and HADOOP-18406 have been backported.
was (Author: simbadzina):
Hi [~tasanuma]. We haven't backported HDFS-13522 and HDFS-16767 to the 3.x line
yet. There were large merge conflicts so we deferred the work.
So far, only HDFS-16669 and HDFS-18406 have been backported.
> HDFS-13522: Add federated nameservices states to client protocol and
> propagate it between routers and clients.
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{{}FederationNamenodeServiceState{}}}.
> This patch captures the state of all namespaces in the routers and propagates
> it to clients. A follow up patch will change router behavior to direct
> requests to the observer.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13522) HDFS-13522: Add federated nameservices states to client protocol and propagate it between routers and clients.
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654691#comment-17654691
]
Simbarashe Dzinamarira commented on HDFS-13522:
---
Hi [~tasanuma]. We haven't backported HDFS-13522 and HDFS-16767 to the 3.x line
yet. There were large merge conflicts so we deferred the work.
So far, only HDFS-16669 and HDFS-18406 have been backported.
> HDFS-13522: Add federated nameservices states to client protocol and
> propagate it between routers and clients.
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png,
> observer_reads_in_rbf_proposal_simbadzina_v1.pdf,
> observer_reads_in_rbf_proposal_simbadzina_v2.pdf
>
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{{}FederationNamenodeServiceState{}}}.
> This patch captures the state of all namespaces in the routers and propagates
> it to clients. A follow up patch will change router behavior to direct
> requests to the observer.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16883) Duplicate field name in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HDFS-16883.
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Duplicate field name in hdfs-default.xml
>
>
> Key: HDFS-16883
> URL: https://issues.apache.org/jira/browse/HDFS-16883
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: YUBI LEE
>Assignee: YUBI LEE
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: image-2023-01-04-10-02-16-881.png
>
>
> {{"dfs.storage.policy.satisfier.enabled"}} and
> "{{{}dfs.storage.policy.satisfier.mode"{}}} is specified in the same
> `property` tag in hdfs-default.xml.
> It should be separated. Because of this, on website, the description is wrong.
> [https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml]
> !image-2023-01-04-10-02-16-881.png|width=1697,height=89!
> {{"dfs.storage.policy.satisfier.enabled"}} is delete since
> https://issues.apache.org/jira/browse/HDFS-13057.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16883) Duplicate field name in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654678#comment-17654678
]
Ayush Saxena commented on HDFS-16883:
-
Committed to trunk.
Thanx [~eub] for the contribution!!!
PS. Added [~eub] as HDFS Contributor to assign the ticket
> Duplicate field name in hdfs-default.xml
>
>
> Key: HDFS-16883
> URL: https://issues.apache.org/jira/browse/HDFS-16883
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: YUBI LEE
>Assignee: YUBI LEE
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2023-01-04-10-02-16-881.png
>
>
> {{"dfs.storage.policy.satisfier.enabled"}} and
> "{{{}dfs.storage.policy.satisfier.mode"{}}} is specified in the same
> `property` tag in hdfs-default.xml.
> It should be separated. Because of this, on website, the description is wrong.
> [https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml]
> !image-2023-01-04-10-02-16-881.png|width=1697,height=89!
> {{"dfs.storage.policy.satisfier.enabled"}} is delete since
> https://issues.apache.org/jira/browse/HDFS-13057.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16883) Duplicate field name in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena reassigned HDFS-16883:
---
Assignee: YUBI LEE
> Duplicate field name in hdfs-default.xml
>
>
> Key: HDFS-16883
> URL: https://issues.apache.org/jira/browse/HDFS-16883
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: YUBI LEE
>Assignee: YUBI LEE
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2023-01-04-10-02-16-881.png
>
>
> {{"dfs.storage.policy.satisfier.enabled"}} and
> "{{{}dfs.storage.policy.satisfier.mode"{}}} is specified in the same
> `property` tag in hdfs-default.xml.
> It should be separated. Because of this, on website, the description is wrong.
> [https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml]
> !image-2023-01-04-10-02-16-881.png|width=1697,height=89!
> {{"dfs.storage.policy.satisfier.enabled"}} is delete since
> https://issues.apache.org/jira/browse/HDFS-13057.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16883) Duplicate field name in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654677#comment-17654677
]
Ayush Saxena commented on HDFS-16883:
-
Committed to trunk.
Thanx [~eub] for the contribution!!!
> Duplicate field name in hdfs-default.xml
>
>
> Key: HDFS-16883
> URL: https://issues.apache.org/jira/browse/HDFS-16883
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: YUBI LEE
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2023-01-04-10-02-16-881.png
>
>
> {{"dfs.storage.policy.satisfier.enabled"}} and
> "{{{}dfs.storage.policy.satisfier.mode"{}}} is specified in the same
> `property` tag in hdfs-default.xml.
> It should be separated. Because of this, on website, the description is wrong.
> [https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml]
> !image-2023-01-04-10-02-16-881.png|width=1697,height=89!
> {{"dfs.storage.policy.satisfier.enabled"}} is delete since
> https://issues.apache.org/jira/browse/HDFS-13057.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16883) Duplicate field name in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654676#comment-17654676
]
ASF GitHub Bot commented on HDFS-16883:
---
ayushtkn merged PR #5271:
URL: https://github.com/apache/hadoop/pull/5271
> Duplicate field name in hdfs-default.xml
>
>
> Key: HDFS-16883
> URL: https://issues.apache.org/jira/browse/HDFS-16883
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: YUBI LEE
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2023-01-04-10-02-16-881.png
>
>
> {{"dfs.storage.policy.satisfier.enabled"}} and
> "{{{}dfs.storage.policy.satisfier.mode"{}}} is specified in the same
> `property` tag in hdfs-default.xml.
> It should be separated. Because of this, on website, the description is wrong.
> [https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml]
> !image-2023-01-04-10-02-16-881.png|width=1697,height=89!
> {{"dfs.storage.policy.satisfier.enabled"}} is delete since
> https://issues.apache.org/jira/browse/HDFS-13057.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-16883) Duplicate field name in hdfs-default.xml
[
https://issues.apache.org/jira/browse/HDFS-16883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena moved HADOOP-18588 to HDFS-16883:
--
Component/s: documentation
(was: documentation)
Key: HDFS-16883 (was: HADOOP-18588)
Project: Hadoop HDFS (was: Hadoop Common)
> Duplicate field name in hdfs-default.xml
>
>
> Key: HDFS-16883
> URL: https://issues.apache.org/jira/browse/HDFS-16883
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: YUBI LEE
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2023-01-04-10-02-16-881.png
>
>
> {{"dfs.storage.policy.satisfier.enabled"}} and
> "{{{}dfs.storage.policy.satisfier.mode"{}}} is specified in the same
> `property` tag in hdfs-default.xml.
> It should be separated. Because of this, on website, the description is wrong.
> [https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml]
> !image-2023-01-04-10-02-16-881.png|width=1697,height=89!
> {{"dfs.storage.policy.satisfier.enabled"}} is delete since
> https://issues.apache.org/jira/browse/HDFS-13057.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16865) RBF: The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC.
[
https://issues.apache.org/jira/browse/HDFS-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654666#comment-17654666
]
ASF GitHub Bot commented on HDFS-16865:
---
ayushtkn commented on code in PR #5200:
URL: https://github.com/apache/hadoop/pull/5200#discussion_r1061936677
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java:
##
@@ -465,6 +465,24 @@ public void setOwner(String src, String username, String
groupname)
}
}
+ /**
+ * Try to get the remote location whose bpId is same with the input bpId
from the input locations.
+ * @param locations the input RemoteLocations.
+ * @param bpId the input bpId.
+ * @return the remote location whose bpId is same with the input.
+ * @throws IOException
+ */
+ private RemoteLocation getLocationWithBPID(List locations,
String bpId)
+ throws IOException {
+String nsId = rpcClient.getNameserviceForBlockPoolId(bpId);
+for (RemoteLocation l : locations) {
+ if (l.getNameserviceId().equals(nsId)) {
+return l;
+ }
+}
+throw new IOException("Can't found remote locations for the " + bpId);
Review Comment:
This throwing exception ain't backward compatible.
Like:
* Create a file
* Add some blocks
* The mount entry gets deleted
* Call complete to close
* Earlier: The file would have got successfully closed, but now it will
throw an exception.
> RBF: The source path is always / after RBF proxied the complete, addBlock and
> getAdditionalDatanode RPC.
>
>
> Key: HDFS-16865
> URL: https://issues.apache.org/jira/browse/HDFS-16865
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> The source path is always / after RBF proxied the complete, addBlock and
> getAdditionalDatanode RPC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16881) Warn if AccessControlEnforcer runs for a long time to check permission
[
https://issues.apache.org/jira/browse/HDFS-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654585#comment-17654585
]
ASF GitHub Bot commented on HDFS-16881:
---
hadoop-yetus commented on PR #5268:
URL: https://github.com/apache/hadoop/pull/5268#issuecomment-1371266019
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 53s | | trunk passed |
| +1 :green_heart: | compile | 1m 44s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 1m 22s |
[/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/3/artifact/out/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 2 new + 910
unchanged - 0 fixed = 912 total (was 910) |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 1m 16s |
[/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/3/artifact/out/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08
with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 2 new + 889
unchanged - 0 fixed = 891 total (was 889) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 385m 4s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 503m 0s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5268 |
| Optional Tests | dupname asflicense compile
[jira] [Commented] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl
[
https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654580#comment-17654580
]
ASF GitHub Bot commented on HDFS-16848:
---
hadoop-yetus commented on PR #5147:
URL: https://github.com/apache/hadoop/pull/5147#issuecomment-1371251060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 40m 38s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5147/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 142m 5s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRBFConfigFields |
| |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5147/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5147 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux f43d32870292 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dd48aa84941af03fb12277bdc1f12a9b4a5e69be |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
[jira] [Commented] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl
[
https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654545#comment-17654545
]
ASF GitHub Bot commented on HDFS-16848:
---
goiri commented on code in PR #5147:
URL: https://github.com/apache/hadoop/pull/5147#discussion_r1061664359
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java:
##
@@ -192,22 +255,45 @@ public boolean putAll(
String znode = getZNodeForClass(recordClass);
long start = monotonicNow();
-boolean status = true;
-for (T record : records) {
- String primaryKey = getPrimaryKey(record);
- String recordZNode = getNodePath(znode, primaryKey);
- byte[] data = serialize(record);
- if (!writeNode(recordZNode, data, update, error)){
-status = false;
+final AtomicBoolean status = new AtomicBoolean(true);
+if (enableConcurrent) {
+ List> callables = new ArrayList<>();
+ records.forEach(record ->
+ callables.add(
+ () -> {
+String primaryKey = getPrimaryKey(record);
+String recordZNode = getNodePath(znode, primaryKey);
+byte[] data = serialize(record);
+if (!writeNode(recordZNode, data, update, error)) {
+ status.set(false);
+}
+return null;
+ }
+ )
+ );
+ try {
+executorService.invokeAll(callables);
+ } catch (Exception e) {
+LOG.error("Write record failed : {}", e.getMessage(), e);
+throw new IOException(e);
}
+} else {
+ records.forEach(record -> {
Review Comment:
Could we just invoke the callables as in the concurrent case but serially?
In that way we would have a single piece of code to create the callables and
then we do the if to invoke it concurrent or serial.
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/store/driver/TestStateStoreZK.java:
##
@@ -126,33 +133,73 @@ private void testGetNullRecord(
assertNull(curatorFramework.checkExists().forPath(znode));
}
+ @Test
+ public void testAsyncPerformance() throws Exception {
+StateStoreZooKeeperImpl stateStoreDriver = (StateStoreZooKeeperImpl)
getStateStoreDriver();
+List insertList = new ArrayList<>();
+for (int i = 0; i < 1000; i++) {
+ MountTable newRecord = generateFakeRecord(MountTable.class);
+ insertList.add(newRecord);
+}
+// Insert Multiple on sync mode
+long startSync = Time.now();
+stateStoreDriver.putAll(insertList, true, false);
+long endSync = Time.now();
+stateStoreDriver.removeAll(MembershipState.class);
+
+stateStoreDriver.setEnableConcurrent(true);
+// Insert Multiple on async mode
+long startAsync = Time.now();
+stateStoreDriver.putAll(insertList, true, false);
+long endAsync = Time.now();
+System.out.printf("Sync mode total running time is %d ms, "
++ "and async mode total running time is %d ms",
+endSync - startSync, endAsync - startAsync);
+assertTrue((endSync - startSync) > (endAsync - startAsync) * 2);
+ }
+
@Test
public void testGetNullRecord() throws Exception {
-testGetNullRecord(getStateStoreDriver());
+StateStoreZooKeeperImpl stateStoreDriver = (StateStoreZooKeeperImpl)
getStateStoreDriver();
+testGetNullRecord(stateStoreDriver);
+stateStoreDriver.setEnableConcurrent(true);
+testGetNullRecord(stateStoreDriver);
}
@Test
public void testInsert()
throws IllegalArgumentException, IllegalAccessException, IOException {
-testInsert(getStateStoreDriver());
+StateStoreZooKeeperImpl stateStoreDriver = (StateStoreZooKeeperImpl)
getStateStoreDriver();
+testInsert(stateStoreDriver);
+stateStoreDriver.setEnableConcurrent(true);
+testInsert(stateStoreDriver);
}
@Test
public void testUpdate()
throws IllegalArgumentException, ReflectiveOperationException,
IOException, SecurityException {
-testPut(getStateStoreDriver());
+StateStoreZooKeeperImpl stateStoreDriver = (StateStoreZooKeeperImpl)
getStateStoreDriver();
+testPut(stateStoreDriver);
+stateStoreDriver.setEnableConcurrent(true);
+testPut(stateStoreDriver);
}
@Test
public void testDelete()
throws IllegalArgumentException, IllegalAccessException, IOException {
-testRemove(getStateStoreDriver());
+StateStoreZooKeeperImpl stateStoreDriver = (StateStoreZooKeeperImpl)
getStateStoreDriver();
+testRemove(stateStoreDriver);
+stateStoreDriver.setEnableConcurrent(true);
Review Comment:
Add a break line to split the concurrent from the other.
> RBF: Improve StateStoreZookeeperImpl
> -
>
> Key: HDFS-16848
>
[jira] [Commented] (HDFS-16865) RBF: The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC.
[
https://issues.apache.org/jira/browse/HDFS-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654542#comment-17654542
]
ASF GitHub Bot commented on HDFS-16865:
---
goiri commented on code in PR #5200:
URL: https://github.com/apache/hadoop/pull/5200#discussion_r1061660594
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java:
##
@@ -979,7 +979,10 @@ public boolean complete(String src, String clientName,
ExtendedBlock last, long fileId)
throws IOException {
checkNNStartup();
-return namesystem.completeFile(src, clientName, last, fileId);
+boolean result = namesystem.completeFile(src, clientName, last, fileId);
+LOG.debug("complete: src={}, clientName={}, fileId={}, result={}.",
Review Comment:
The others use somethine like:
```
blockStateChangeLog.debug("*BLOCK*
```
Should we use something similar?
> RBF: The source path is always / after RBF proxied the complete, addBlock and
> getAdditionalDatanode RPC.
>
>
> Key: HDFS-16865
> URL: https://issues.apache.org/jira/browse/HDFS-16865
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
>
> The source path is always / after RBF proxied the complete, addBlock and
> getAdditionalDatanode RPC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16865) RBF: The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC.
[ https://issues.apache.org/jira/browse/HDFS-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-16865: --- Summary: RBF: The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. (was: The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC.) > RBF: The source path is always / after RBF proxied the complete, addBlock and > getAdditionalDatanode RPC. > > > Key: HDFS-16865 > URL: https://issues.apache.org/jira/browse/HDFS-16865 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: ZanderXu >Assignee: ZanderXu >Priority: Major > Labels: pull-request-available > > The source path is always / after RBF proxied the complete, addBlock and > getAdditionalDatanode RPC. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16880) modify invokeSingleXXX interface in order to pass actual file src to namenode for debug info.
[
https://issues.apache.org/jira/browse/HDFS-16880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654534#comment-17654534
]
ASF GitHub Bot commented on HDFS-16880:
---
goiri commented on code in PR #5262:
URL: https://github.com/apache/hadoop/pull/5262#discussion_r1061651607
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcServer.java:
##
@@ -687,7 +687,7 @@ T invokeAtAvailableNs(RemoteMethod method, Class
clazz)
// If default Ns is present return result from that namespace.
if (!nsId.isEmpty()) {
try {
-return rpcClient.invokeSingle(nsId, method, clazz);
+return rpcClient.invokeSingle(nsId, method, clazz, "");
Review Comment:
I meant to make it part of RemoteMethod.
> modify invokeSingleXXX interface in order to pass actual file src to namenode
> for debug info.
> -
>
> Key: HDFS-16880
> URL: https://issues.apache.org/jira/browse/HDFS-16880
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.3.4
>Reporter: ZhangHB
>Priority: Major
> Labels: pull-request-available
>
> We found lots of INFO level log like below:
> {quote}2022-12-30 15:31:04,169 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile: / is closed by
> DFSClient_attempt_1671783180362_213003_m_77_0_1102875551_1
> 2022-12-30 15:31:04,186 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile: / is closed by DFSClient_NONMAPREDUCE_1198313144_27480
> {quote}
> It lost the real path of completeFile. Actually this is caused by :
>
> *org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient#invokeSingle(java.lang.String,
> org.apache.hadoop.hdfs.server.federation.router.RemoteMethod)*
> In this method, it instantiates a RemoteLocationContext object:
> *RemoteLocationContext loc = new RemoteLocation(nsId, "/", "/");*
> and then execute: *Object[] params = method.getParams(loc);*
> The problem is right here, becasuse we always use new RemoteParam(), so,
> context.getDest() always return "/"; That's why we saw lots of incorrect logs.
>
> After diving into invokeSingleXXX source code, I found the following RPCs
> classified as need actual src and not need actual src.
>
> *need src path RPC:*
> addBlock、abandonBlock、getAdditionalDatanode、complete
> *not need src path RPC:*
> updateBlockForPipeline、reportBadBlocks、getBlocks、updatePipeline、invokeAtAvailableNs（invoked
> by:
> getServerDefaults、getBlockKeys、getTransactionID、getMostRecentCheckpointTxId、versionRequest、getStoragePolicies）
>
> After changes, the src can be pass to NN correctly.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16882) RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath
[ https://issues.apache.org/jira/browse/HDFS-16882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-16882: --- Description: Currently, the default value of "dfs.federation.router.mount-table.cache.enable" is true and the default value of "dfs.federation.router.mount-table.max-cache-size" is 1. But there is no metric that display cache hit rate, I think we can add a hit rate metric to watch the Cache performance and better tuning the parameters. was: Currently, the default value of "dfs.federation.router.mount-table.cache.enable" is ture， the default value of "dfs.federation.router.mount-table.max-cache-size" is 1. But there is no metric that display cache hit rate, I think we can add a hit rate metric to watch the Cache performance and better tuning the parameters. > RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath > -- > > Key: HDFS-16882 > URL: https://issues.apache.org/jira/browse/HDFS-16882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.3.4 >Reporter: ZhangHB >Priority: Minor > > Currently, the default value of > "dfs.federation.router.mount-table.cache.enable" is true and the default > value of "dfs.federation.router.mount-table.max-cache-size" is 1. > But there is no metric that display cache hit rate, I think we can add a hit > rate metric to watch the Cache performance and better tuning the parameters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16882) RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath
[ https://issues.apache.org/jira/browse/HDFS-16882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-16882: --- Summary: RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath (was: Add cache hit rate metric in MountTableResolver#getDestinationForPath) > RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath > -- > > Key: HDFS-16882 > URL: https://issues.apache.org/jira/browse/HDFS-16882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.3.4 >Reporter: ZhangHB >Priority: Minor > > Currently, the default value of > "dfs.federation.router.mount-table.cache.enable" is ture， > the default value of "dfs.federation.router.mount-table.max-cache-size" is > 1. > But there is no metric that display cache hit rate, I think we can add a hit > rate metric to watch the Cache performance and better tuning the parameters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl
[
https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654492#comment-17654492
]
ASF GitHub Bot commented on HDFS-16848:
---
hadoop-yetus commented on PR #5147:
URL: https://github.com/apache/hadoop/pull/5147#issuecomment-1371028638
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5147/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 41m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5147/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 143m 2s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRBFConfigFields |
| | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5147/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5147 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 9f9319d44586 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 14a6ad0b798326fc269c39f0afe081259ba1fbf6 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-16848) RBF: Improve StateStoreZookeeperImpl
[
https://issues.apache.org/jira/browse/HDFS-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1765#comment-1765
]
ASF GitHub Bot commented on HDFS-16848:
---
howzi commented on code in PR #5147:
URL: https://github.com/apache/hadoop/pull/5147#discussion_r1061436260
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/driver/impl/StateStoreZooKeeperImpl.java:
##
@@ -63,8 +72,17 @@ public class StateStoreZooKeeperImpl extends
StateStoreSerializableImpl {
RBFConfigKeys.FEDERATION_STORE_PREFIX + "driver.zk.";
public static final String FEDERATION_STORE_ZK_PARENT_PATH =
FEDERATION_STORE_ZK_DRIVER_PREFIX + "parent-path";
+ public static final String FEDERATION_STORE_ZK_CLIENT_THREADS_SIZE =
+ FEDERATION_STORE_ZK_DRIVER_PREFIX + "client.size";
+ public static final int FEDERATION_STORE_ZK_CLIENT_THREADS_SIZE_DEFAULT = 10;
+ public static final String FEDERATION_STORE_ZK_CLIENT_CONCURRENT =
+ FEDERATION_STORE_ZK_DRIVER_PREFIX + "client.concurrent";
+ public static final boolean FEDERATION_STORE_ZK_CLIENT_CONCURRENT_DEFAULT =
false;
Review Comment:
Thank you very much for your help! Just fixed all above problems, please
check it again~
> RBF: Improve StateStoreZookeeperImpl
> -
>
> Key: HDFS-16848
> URL: https://issues.apache.org/jira/browse/HDFS-16848
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Reporter: Sun Hao
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> Currently, router is getting/updating state from zk sequentially. It will
> slowdown router load/update state cache especially for a large cluster or a
> multi region cluster.
> We propose adding a threadpool to deal with zk state synchronization。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16881) Warn if AccessControlEnforcer runs for a long time to check permission
[
https://issues.apache.org/jira/browse/HDFS-16881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17654362#comment-17654362
]
ASF GitHub Bot commented on HDFS-16881:
---
hadoop-yetus commented on PR #5268:
URL: https://github.com/apache/hadoop/pull/5268#issuecomment-1370669841
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 1m 23s |
[/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/2/artifact/out/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 2 new + 910
unchanged - 0 fixed = 912 total (was 910) |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| -1 :x: | javac | 1m 14s |
[/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/2/artifact/out/results-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08
with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 2 new + 889
unchanged - 0 fixed = 891 total (was 889) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 55s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 364 unchanged
- 0 fixed = 365 total (was 364) |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 356m 5s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5268/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 470m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
