[
https://issues.apache.org/jira/browse/ACCUMULO-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser resolved ACCUMULO-3351.
----------------------------------
Resolution: Fixed
> Tracer can't write traces after offline and online of trace table
> -----------------------------------------------------------------
>
> Key: ACCUMULO-3351
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3351
> Project: Accumulo
> Issue Type: Bug
> Components: trace
> Affects Versions: 1.6.1
> Reporter: Josh Elser
> Assignee: Josh Elser
> Fix For: 1.6.2, 1.7.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> While running tests for ACCUMULO-3167, I updated one of the tests to offline
> the trace table to reduce the possibility that any active logs for the trace
> table would exist in the metadata table.
> A later test went to validate that traces were found for some conditional
> update sessions and hung indefinitely.
> Inspecting the tracer log, the batchwriter had two exceptions due to the
> trace table being offline (as expected), but never recovered when the trace
> table came back online.
> {noformat}
> 2014-11-20 13:08:28,717 [impl.TabletServerBatchWriter] DEBUG: Table trace
> (in) is offline
> org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is
> offline
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> 2014-11-20 13:08:28,720 [tracer.TraceServer] WARN : Problem flushing traces,
> resetting writer. Set log level to DEBUG to see stacktrace. cause:
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint
> violations : 0 security codes: {} # server errors 0 # exceptions 1
> 2014-11-20 13:08:28,720 [tracer.TraceServer] DEBUG: flushing traces failed
> due to exception
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint
> violations : 0 security codes: {} # server errors 0 # exceptions 1
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.flush(TabletServerBatchWriter.java:331)
> at
> org.apache.accumulo.core.client.impl.BatchWriterImpl.flush(BatchWriterImpl.java:59)
> at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:245)
> at
> org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> at
> org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace
> (in) is offline
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> ... 2 more
> 2014-11-20 13:08:28,722 [tracer.TraceServer] WARN : Unable to create a batch
> writer, will retry. Set log level to DEBUG to see stacktrace. cause:
> org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is
> offline
> 2014-11-20 13:08:28,722 [tracer.TraceServer] DEBUG: batch writer creation
> failed with exception.
> org.apache.accumulo.core.client.TableOfflineException: Table trace (in) is
> offline
> at
> org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:86)
> at
> org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
> at
> org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:262)
> at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
> at
> org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> at
> org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> 2014-11-20 13:08:28,723 [tracer.TraceServer] WARN : Problem closing batch
> writer. Set log level to DEBUG to see stacktrace. cause:
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint
> violations : 0 security codes: {} # server errors 0 # exceptions 1
> 2014-11-20 13:08:28,723 [tracer.TraceServer] DEBUG: batch writer close failed
> with exception
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint
> violations : 0 security codes: {} # server errors 0 # exceptions 1
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:354)
> at
> org.apache.accumulo.core.client.impl.BatchWriterImpl.close(BatchWriterImpl.java:54)
> at
> org.apache.accumulo.tracer.TraceServer.resetWriter(TraceServer.java:271)
> at org.apache.accumulo.tracer.TraceServer.flush(TraceServer.java:250)
> at
> org.apache.accumulo.tracer.TraceServer.access$300(TraceServer.java:78)
> at org.apache.accumulo.tracer.TraceServer$1.run(TraceServer.java:235)
> at
> org.apache.accumulo.server.util.time.SimpleTimer$LoggingTimerTask.run(SimpleTimer.java:42)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: org.apache.accumulo.core.client.TableOfflineException: Table trace
> (in) is offline
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:662)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:694)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:233)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addFailedMutations(TabletServerBatchWriter.java:551)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.access$700(TabletServerBatchWriter.java:101)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$FailedMutations.run(TabletServerBatchWriter.java:603)
> ... 2 more
> 2014-11-20 13:10:12,929 [tracer.TraceServer] WARN : writer is not ready;
> discarding span.
> 2014-11-20 13:10:12,930 [tracer.TraceServer] WARN : writer is not ready;
> discarding span.
> {noformat}
> "writer is not ready; discarding span." repeats indefinitely.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
