[ https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571564#comment-17571564 ]
Simbarashe Dzinamarira edited comment on HDFS-13522 at 7/26/22 6:01 PM: ------------------------------------------------------------------------ Thanks [~xuzq_zander] for listing the considerations. When there are 100+ nameservices then the large RPC header would indeed be a problem. However, for a few nameservices, I've assumed the larger RPC header is a better tradeoff that the extra msync call for strong consistency. *We can do a combination of both Design A and B.* So Design B (always msync) would be the baseline which doesn't requirement any client changes. Then if the number of nameservices is under a certain threshold, the router can send extra information in the header. An old client ignores this information, which results in Design B behavior, but a new client transparently forwards this back to the router allowing the router to avoid the msync (Design A). My implementation already does follow Design B to support old clients, and a threshold can be easily added so that Design A is only used in situations where there aren't too many namespaces. [~zhengchenyu] can you share how I can access the slack channel. Is there a slack instance of Apache? was (Author: simbadzina): Thanks [~xuzq_zander] for listing the considerations. When there are 100+ nameservices then the large RPC header would indeed be a problem. However, for a few nameservices, I've assumed the larger RPC header is a better tradeoff that the extra msync call for strong consistency. We can do a combination of both Design A and B. So Design B (always msync) would be the baseline which doesn't requirement any client changes. Then if the number of nameservices is under a certain threshold, the router can send extra information in the header. An old client ignores this information, which results in Design B behavior, but a new client transparently forwards this back to the router allowing the router to avoid the msync (Design A). My implementation already does follow Design B to support old clients, and a threshold can be easily added so that Design A is only used in situations where there aren't too many namespaces. [~zhengchenyu] can you share how I can access the slack channel. Is there a slack instance of Apache? > RBF: Support observer node from Router-Based Federation > ------------------------------------------------------- > > Key: HDFS-13522 > URL: https://issues.apache.org/jira/browse/HDFS-13522 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation, namenode > Reporter: Erik Krogen > Assignee: Simbarashe Dzinamarira > Priority: Major > Labels: pull-request-available > Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch, > HDFS-13522_WIP.patch, HDFS-13522_proposal_zhengchenyu_v1.pdf, RBF_ Observer > support.pdf, Router+Observer RPC clogging.png, > ShortTerm-Routers+Observer.png, > observer_reads_in_rbf_proposal_simbadzina_v1.pdf, > observer_reads_in_rbf_proposal_simbadzina_v2.pdf > > Time Spent: 20h > Remaining Estimate: 0h > > Changes will need to occur to the router to support the new observer node. > One such change will be to make the router understand the observer state, > e.g. {{FederationNamenodeServiceState}}. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org