[
https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359009#comment-14359009
]
Steve Loughran commented on HADOOP-11708:
-----------------------------------------
hmmm.
# HADOOP-9361 skipped on the output stream stuff in the rush to get things into
hadoop2.5, and for all its stuff is very vague about concurrency guarantees. As
it extends {{java.io.OutputStream}}, the real concurrency semantics should be
defined there. And, looking at that openjdk code, it appears to be "no
guarantees". HBase have some expectations that may already be invalid.
# We've also seen from HDFS-6735 that HBase likes non-blocking reads for
performance enhancements...they clearly have different/needs expectations.
HBASE-8722 is where someone proposed writing them down. If you look at
HDFS-6735 we had to spend a lot of time thinking about & looking at what is
going on, trying to understand the implicit concurrency guarantees of HDFS &
the expectations of code as seen by IDE-based scans for usage.
# If you are proposing we define those concurrency semantics more strictly, in
the filesystem specification and the code that would be great: someone did need
to sit down and define the APIs as based on HDFS behaviour. This is not
something we can rush into in the last minute though. I also worry about the
proposed culling of +10 sync blocks, especially on the basis that "they appear
to be unneeded". That's the kind of question that needs to be answered by
looking at the git/svn history of those lines & correlating them with past
JIRAs, before doing what could potentially cause problems. And, at write()
time, that means data-loss/corruption problems beyond just encrypted data.
Short term? I'd say note the concurrency expectations of HBase and an output
stream & come up with a change for the CryptoOutputStream which implements
consistent concurrency semantics. The runup to a release is not the time to
change something so foundational.
Longer term? Decide what the concurrency guarantees should be, scan through the
core code stack to identify risky uses of flush() (actually I'd add log/count
to DFSOutputStream & see if we could detect & log re-entrant ops across
threads: flush/write overlap., flush/flush concurrency, hflush+write, ...). As
with HADOOP-9361, HDFS defines the defacto semantics, but its the uses of that
code which really set the expectations of applications. Here we've just found
HBases
> CryptoOutputStream synchronization differences from DFSOutputStream break
> HBase
> -------------------------------------------------------------------------------
>
> Key: HADOOP-11708
> URL: https://issues.apache.org/jira/browse/HADOOP-11708
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.6.0
> Reporter: Sean Busbey
> Assignee: Sean Busbey
> Priority: Critical
>
> For the write-ahead-log, HBase writes to DFS from a single thread and sends
> sync/flush/hflush from a configurable number of other threads (default 5).
> FSDataOutputStream does not document anything about being thread safe, and it
> is not thread safe for concurrent writes.
> However, DFSOutputStream is thread safe for concurrent writes + syncs. When
> it is the stream FSDataOutputStream wraps, the combination is threadsafe for
> 1 writer and multiple syncs (the exact behavior HBase relies on).
> When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted
> between FSDataOutputStream and DFSOutputStream. It is proactively labeled as
> not thread safe, and this composition is not thread safe for any operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
