[ https://issues.apache.org/jira/browse/HDFS-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622948#comment-16622948 ]
Chen Liang commented on HDFS-13873: ----------------------------------- Thanks [~csun]! I had some thoughts on this, sharing here for references: version 1: have a tracker that, whenever client sends request to Observer, trackers records Observer's current state id X, and timestamp tx comparing to previous value Y and previous timestamp ty and t = (X - Y) / (tx - ty) gives an estimation of how long it takes for observer to proceed one txid. (this can be measured as moving average for better accuracy). And say delta = clientStateId - X, then delta * t gives the estimate time of when the client request can start being processed i.e. the msync wait time. Plan 2: instead of tracking Observer state id increasing rate. We could also have t = the average time of processing one request. (This needs more code to measure time spent for a request to be in the queue until finished). The delta * t then becomes the estimate of when the client request will actually finish. version 2 requires more code changes, but is able to handle the case that, Observer state id is actually not too far behind, but Observer node itself is being too slow, causing still a long processing time of a request. Which is not captured by version 1. The downside though, it seemed to me there can be cases where version 2 can reject many calls over-aggressively. Also addressing slow Observer seems a bit beyond the scope of this Jira. I would say maybe we can go with the simpler of version 1 first and see how it works out. Any comments [~csun], [~shv]? > ObserverNode should reject read requests when it is too far behind. > ------------------------------------------------------------------- > > Key: HDFS-13873 > URL: https://issues.apache.org/jira/browse/HDFS-13873 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode > Affects Versions: HDFS-12943 > Reporter: Konstantin Shvachko > Assignee: Chao Sun > Priority: Major > > Add a server-side threshold for ObserverNode to reject read requests when it > is too far behind. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org