[ https://issues.apache.org/jira/browse/CASSANDRA-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sam Overton reopened CASSANDRA-3677: ------------------------------------ I've been able to reproduce this in 1.07 and in trunk as follows: * create a cluster of 2 nodes * stop one of the nodes * insert some data at RF=1, CL=ANY - this causes hints to be stored * nodetool removetoken on the token of the dead node * some time up to 10 minutes later the exception is logged: ERROR [OptionalTasks:1] 2012-02-07 14:41:57,710 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[OptionalTasks:1,5,main] java.lang.NullPointerException at org.cliffc.high_scale_lib.NonBlockingHashMap.hash(NonBlockingHashMap.java:113) at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:553) at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:348) at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfAbsent(NonBlockingHashMap.java:319) at org.cliffc.high_scale_lib.NonBlockingHashSet.add(NonBlockingHashSet.java:32) at org.apache.cassandra.db.HintedHandOffManager.scheduleHintDelivery(HintedHandOffManager.java:410) at org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:395) at org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:119) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Adverse Effects: * The 10min repeating scheduleAllDeliveries() introduced in CASSANDRA-3554 will no longer fire (since the exception was uncaught), so the node has now regressed to pre-CASSANDRA-3554 behaviour * Even after the node is restarted, the exception will be thrown again next time the schedule is run. * This will continue until the row tombstone for the dead node's hints is garbage-collected (10 days) Fix: * don't try to schedule delivery of hints for tokens which are no longer valid endpoints (see attached). > NPE during HH delivery when gossip turned off on target > ------------------------------------------------------- > > Key: CASSANDRA-3677 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3677 > Project: Cassandra > Issue Type: Bug > Affects Versions: 1.0.7 > Reporter: Radim Kolar > Assignee: Brandon Williams > Priority: Trivial > Attachments: 3677.txt > > > probably not important bug > ERROR [OptionalTasks:1] 2011-12-27 21:44:25,342 AbstractCassandraDaemon.java > (line 138) Fatal exception in thread Thread[OptionalTasks:1,5,main] > java.lang.NullPointerException > at > org.cliffc.high_scale_lib.NonBlockingHashMap.hash(NonBlockingHashMap.java:113) > at > org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:553) > at > org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:348) > at > org.cliffc.high_scale_lib.NonBlockingHashMap.putIfAbsent(NonBlockingHashMap.java:319) > at > org.cliffc.high_scale_lib.NonBlockingHashSet.add(NonBlockingHashSet.java:32) > at > org.apache.cassandra.db.HintedHandOffManager.scheduleHintDelivery(HintedHandOffManager.java:371) > at > org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:356) > at > org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:84) > at > org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:119) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira