[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767613#comment-16767613
]
Erik Krogen commented on HDFS-14272:
------------------------------------
This problem is harder than I thought.
Initially I tried to add a flag, {{activeNameNodeContacted}}, which starts as
false and it set to true once the {{activeProxy}} is used. If this is false, I
bypassed the observer-read logic and went straight to the {{activeProxy}},
assuming that this would contact an Active NN. Unfortunately, the active proxy
may not actually contact an active -- it will happily go to an observer node if
the operation is a read, or to any node if it the operation is uncoordinated
(e.g. {{getHAServiceState()}}).
My next thought was to inject an {{msync()}} call the first time that the
{{ObserverReadInvocationHandler}} is used. However, currently the observer will
service an {{msync()}} -- {{activeOnly}} is true, so the {{activeProxy}} is
used, but it isn't enforced by the server (see below), so the active proxy may
still use an observer:
{code:name=NameNodeRpcServer.java}
@Override // ClientProtocol
public void msync() throws IOException {
// TODO : need to be filled up if needed. May be a no-op here.
}
{code}
I think in HDFS-14250 when we marked {{msync()}} as {{activeOnly}}, we should
have also added server-side enforcement that only the active would service
{{msync()}}. Then the above works.
I've attached a v000 patch making the changes described above: add an
{{msync()}} call on the first invocation of {{ObserverReadInvocationHandler}},
and enhance {{msync()}} to properly reject calls not made to the active. I
expect some other Observer-related tests will fail due to these changes, but
posting this for now for others to take a look.
> [SBN read] HDFS command line tools does not guarantee consistency
> -----------------------------------------------------------------
>
> Key: HDFS-14272
> URL: https://issues.apache.org/jira/browse/HDFS-14272
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
> SSL + Kerberos + RPC encryption
> Reporter: Wei-Chiu Chuang
> Assignee: Erik Krogen
> Priority: Major
> Attachments: HDFS-14272.000.patch
>
>
> It is typical for integration tests to create some files and then check their
> existence. For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with
> Consistent Standby Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the
> first one, is not aware of the state id returned from the first bash command.
> So ObserverNode wouldn't wait for the the edits to get propagated, and thus
> fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and
> this becomes very annoying. (I am still trying to figure out why this
> Observer has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
