D M Murali Krishna Reddy created HDFS-18028:
-----------------------------------------------
Summary: Fix Read Consistency issues from Observer Name Node
Key: HDFS-18028
URL: https://issues.apache.org/jira/browse/HDFS-18028
Project: Hadoop HDFS
Issue Type: Bug
Reporter: D M Murali Krishna Reddy
With Observer reads enabled, data loss has been observed for a few Spark jobs
on heavily loaded clusters.
The cause appears to be Observer lag in applying edits: from the audit logs,
the gap between the write from a Spark executor (task commit rename on the
Active) and the read from the Spark driver (listStatus served by the Observer)
was ~5 ms, and the Observer had not yet applied the rename. The driver silently
received a stale listing, the missing task output was dropped during job
commit, and the job still succeeded. Details and reproduction:
https://lists.apache.org/thread.html/wbrd50sr83fv6b52bzt51qwvwqlfnpkd (see also
[SPARK-36810|https://issues.apache.org/jira/browse/SPARK-36810]).
One mitigation is to set dfs.client.failover.observer.auto-msync-period=0 for
the driver, but this sends an msync before every read and would cut into the
performance gains expected from Observer reads.
Another is to update FileOutputCommitter to msync() before the listStatus in
commitJob() where this race was observed, but that fixes only one code path;
any other application where one client reads what another client just wrote has
the same exposure.
Raising this to discuss whether HDFS can provide a stronger default guarantee
for cross-client read-after-write, rather than relying on every application to
opt in.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]