Ted Yu created HBASE-11620:
------------------------------
Summary: Propagate decoder exception to HLogSplitter so that loss
of data is avoided
Key: HBASE-11620
URL: https://issues.apache.org/jira/browse/HBASE-11620
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Priority: Critical
Reported by Kiran in this thread: "HBase file encryption, inconsistencies
observed and data loss"
After step 4 ( i.e disabling of WAL encryption, removing
SecureProtobufReader/Writer and restart), read of encrypted WAL fails mainly
due to EOF exception at Basedecoder. This is not considered as error and these
WAL are being moved to /oldWALs.
Following is observed in log files:
{code}
2014-07-30 19:44:29,254 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
wal.HLogSplitter: Splitting hlog:
hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017,
length=172
2014-07-30 19:44:29,254 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
wal.HLogSplitter: DistributedLogReplay = false
2014-07-30 19:44:29,313 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
util.FSHDFSUtils: Recovering lease on dfs file
hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017
2014-07-30 19:44:29,315 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
util.FSHDFSUtils: recoverLease=true, attempt=0 on
file=hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017
after 1ms
2014-07-30 19:44:29,429 DEBUG [RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-0]
wal.HLogSplitter: Writer thread
Thread[RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-0,5,main]: starting
2014-07-30 19:44:29,429 DEBUG [RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-1]
wal.HLogSplitter: Writer thread
Thread[RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-1,5,main]: starting
2014-07-30 19:44:29,430 DEBUG [RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-2]
wal.HLogSplitter: Writer thread
Thread[RS_LOG_REPLAY_OPS-HOST-16:15264-1-Writer-2,5,main]: starting
2014-07-30 19:44:29,591 ERROR [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
codec.BaseDecoder: Partial cell read caused by EOF: java.io.IOException:
Premature EOF from inputStream
2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
wal.HLogSplitter: Finishing writing output logs and closing down.
2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
wal.HLogSplitter: Waiting for split writer threads to finish
2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
wal.HLogSplitter: Split writers finished
2014-07-30 19:44:29,592 INFO [RS_LOG_REPLAY_OPS-HOST-16:15264-1]
wal.HLogSplitter: Processed 0 edits across 0 regions; log
file=hdfs://HOST-16:18020/hbase/WALs/HOST-16,15264,1406725441997-splitting/HOST-16%2C15264%2C1406725441997.1406725444017
is corrupted = false progress failed = false
{code}
To fix this, we need to propagate EOF exception to HLogSplitter. Any
suggestions on the fix?
-------- (end of quote from Kiran)
In BaseDecoder#rethrowEofException() :
{code}
if (!isEof) throw ioEx;
LOG.error("Partial cell read caused by EOF: " + ioEx);
EOFException eofEx = new EOFException("Partial cell read");
eofEx.initCause(ioEx);
throw eofEx;
{code}
throwing EOFException would not propagate the "Partial cell read" condition to
HLogSplitter which doesn't treat EOFException as an error.
I think a new exception type (DecoderException e.g.) should be used above.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
