[
https://issues.apache.org/jira/browse/CASSANDRA-12844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819815#comment-15819815
]
Harikrishnan edited comment on CASSANDRA-12844 at 1/12/17 1:33 AM:
-------------------------------------------------------------------
Hi,
We reproduced this two times , we were trying to bring down a node by issuing
nodetool drain. One interesting aspect is there were lot mutation drops and
hint replay was happening to most of the nodes while drain is being issued.Will
try to reproduce it again .
was (Author: hari708):
Hi,
We reproduced this two times , we were trying to bring down a node by issuing
nodetool drain. One interesting aspect is there were lot mutation drops and
hint replay was happening to most of the nodes while drain is being issued.
> nodetool drain causing mutiple nodes crashing with hint file corruption in
> Cassandra 3.9
> ----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-12844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12844
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Harikrishnan
> Priority: Critical
> Labels: hints
>
> The steps are as follows.
> we have 4/4 node cassandra running in 3.9 version.
> In one node made some changes to cassanra.yaml. issued a nodetool drain
> killed the cassandra process and restarted the node. After sometime nodetool
> status reported multiple nodes are down in that DC.
> Went and check the system.log of all the files and found the hint corruption
> occuring(CASSANDRA-12728). nodetool drain causing this corruption and
> bringing multiple nodes down is a big concern.
> ERROR [HintsDispatcher:2] 2016-10-26 12:17:59,361
> HintsDispatchExecutor.java:225 - Failed to dispatch hints file
> 4d1362f0-053c-4042-80a7-bfc85a26c90f-1477509190999-1.hints: file is corrupted
> ({})
> org.apache.cassandra.io.FSReadError: java.io.EOFException
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259)
> [apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242)
> [apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220)
> [apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199)
> [apache-cassandra-3.9.jar:3.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_102]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_102]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_102]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
