[
https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HDFS-10702:
---------------------------------
Attachment: HDFS-10797.006.patch
I had a conversation with [~clouderajiayi] about taking this over - meant to
post that previously, my apologies.
Attaching a patch with substantially rewritten logic in the ProxyProvider.
Previously the 'active' was determined by which request completed first, but we
can't rely on the same logic as RequestHedgingProxyProvider because when stale
reads are enabled, any of the NameNodes can respond to virtually any read
request. On the first request and then in the event of failover, it will now
probe all the proxies to determine the active and running standby's, shuffle
the standby's, and then try the first one. In the event of failure, it will try
the next standby in order. When they have all been exhausted it will re-probe
and just use the master. One thing that needs a bit of further testing in
particular is the probing to identify the master and standby nodes. It's a bit
of a hack using the return value of getting the transaction ID, but we don't
want a write operation that modifies anything, and a normal read operation
could almost certainly be served just as well by any of the NameNodes,
regardless of state. Open to other suggestions on this...
This patch also addresses some (but not all, yet) of the remaining feedback
from [~andrew.wang]. One particular point I'd like to discuss is that you
suggest not having a SyncInfo object to encapsulate the transaction ID when it
could be requested separately, but you do suggest having a separate struct to
encapsulate that info in the subsequent request. It does make sense, since it's
cleaner to add getSyncInfo later if we need it than to replace the transaction
ID in the proto later, but I just thought I'd ask: do you think it's likely
instead of just possible there will be additional sync info in the future?
Because if so, we may as well just add it now. Otherwise I'll just go ahead
with your original suggestion...
> Add a Client API and Proxy Provider to enable stale read from Standby
> ---------------------------------------------------------------------
>
> Key: HDFS-10702
> URL: https://issues.apache.org/jira/browse/HDFS-10702
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jiayi Zhou
> Assignee: Jiayi Zhou
> Priority: Minor
> Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch,
> HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch,
> HDFS-10797.006.patch, StaleReadfromStandbyNN.pdf
>
>
> Currently, clients must always talk to the active NameNode when performing
> any metadata operation, which means active NameNode could be a bottleneck for
> scalability. One way to solve this problem is to send read-only operations to
> Standby NameNode. The disadvantage is that it might be a stale read.
> Here, I'm thinking of adding a Client API to enable/disable stale read from
> Standby which gives Client the power to set the staleness restriction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
