[
https://issues.apache.org/jira/browse/ACCUMULO-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157353#comment-14157353
]
Josh Elser commented on ACCUMULO-3182:
--------------------------------------
It appears that I incorrectly assumed that 1.5 would also have this issue in
the first place. Best as I can tell it does not. It seems like this was
introduced with some of the crypto-related changes.
In 1.5, if we can't read the header or there is no header (was this previously
the case for WALs?), we'll return an InputStream seek'ed to '0'. Then, the
LogSorter will attempt to read pairs of LogFileKeys and LogFileValues which
handles an EOFException. In the catch, it will also create the empty MapFile so
that recovery will gracefully complete as well.
> Empty or partial WAL header blocks successful recovery
> ------------------------------------------------------
>
> Key: ACCUMULO-3182
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3182
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Affects Versions: 1.6.1
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: 1.6.2, 1.7.0
>
> Attachments:
> 0001-ACCUMULO-3182-Gracefully-handles-incomplete-missing-.patch
>
>
> Haven't ever seen this one before. A replication IT failed -- looking into
> it, it was because the tserver that came up (after killing the original)
> failed to complete recovery. The below happened a few times before the test
> ultimately timed out.
> {noformat}
> 2014-09-29 04:46:10,259 [zookeeper.DistributedWorkQueue] DEBUG: Looking for
> work in /accumulo/f98e79c4-9dcd-4fb0-8ec9-5804f0818839/recovery
> 2014-09-29 04:46:10,340 [zookeeper.DistributedWorkQueue] DEBUG: got lock for
> af53bf1e-c293-463b-b4de-5efdb8b34962
> 2014-09-29 04:46:10,341 [log.LogSorter] DEBUG: Sorting
> file:/.../test/target/mini-tests/org.apache.accumulo.test.replication.UnorderedWorkAssignerReplicationIT_dataReplicatedToCorrectTableWithoutDrain/accumulo/wal/juno+49195/af53bf1e-c293-463b-b4de-5efdb8b34962
> to
> file:/.../test/target/mini-tests/org.apache.accumulo.test.replication.UnorderedWorkAssignerReplicationIT_dataReplicatedToCorrectTableWithoutDrain/accumulo/recovery/af53bf1e-c293-463b-b4de-5efdb8b34962
> using sortId af53bf1e-c293-463b-b4de-5efdb8b34962
> 2014-09-29 04:46:10,341 [log.LogSorter] INFO : Copying
> file:/var/lib/jenkins/home/jobs/Accumulo-Master-Integration-Tests/workspace/test/target/mini-tests/org.apache.accumulo.test.replication.UnorderedWorkAssignerReplicationIT_dataReplicatedToCorrectTableWithoutDrain/accumulo/wal/juno+49195/af53bf1e-c293-463b-b4de-5efdb8b34962
> to
> file:/.../test/target/mini-tests/org.apache.accumulo.test.replication.UnorderedWorkAssignerReplicationIT_dataReplicatedToCorrectTableWithoutDrain/accumulo/recovery/af53bf1e-c293-463b-b4de-5efdb8b34962
> 2014-09-29 04:46:10,345 [log.LogSorter] ERROR: java.io.EOFException
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at
> org.apache.accumulo.tserver.log.DfsLogger.readHeaderAndReturnStream(DfsLogger.java:282)
> at
> org.apache.accumulo.tserver.log.LogSorter$LogProcessor.sort(LogSorter.java:113)
> at
> org.apache.accumulo.tserver.log.LogSorter$LogProcessor.process(LogSorter.java:93)
> at
> org.apache.accumulo.server.zookeeper.DistributedWorkQueue$1.run(DistributedWorkQueue.java:105)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:745)
> 2014-09-29 04:46:10,346 [log.LogSorter] ERROR: Error during cleanup sort/copy
> af53bf1e-c293-463b-b4de-5efdb8b34962
> java.lang.NullPointerException
> at
> org.apache.accumulo.tserver.log.LogSorter$LogProcessor.close(LogSorter.java:183)
> at
> org.apache.accumulo.tserver.log.LogSorter$LogProcessor.sort(LogSorter.java:151)
> at
> org.apache.accumulo.tserver.log.LogSorter$LogProcessor.process(LogSorter.java:93)
> at
> org.apache.accumulo.server.zookeeper.DistributedWorkQueue$1.run(DistributedWorkQueue.java:105)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
