[ https://issues.apache.org/jira/browse/CASSANDRA-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-2618: ---------------------------------------- Attachment: 2618.txt Patch to avoid the race between add() and clear(). > DynamicSnitch race in adding latencies > -------------------------------------- > > Key: CASSANDRA-2618 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2618 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 0.7.0 > Reporter: Brandon Williams > Assignee: Brandon Williams > Attachments: 2618.txt > > > ERROR 15:33:48,614 Fatal exception in thread Thread[ReadStage:264,5,main] > java.lang.RuntimeException: java.util.NoSuchElementException > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.util.NoSuchElementException > at > java.util.concurrent.LinkedBlockingDeque.removeFirst(LinkedBlockingDeque.java:401) > at > java.util.concurrent.LinkedBlockingDeque.remove(LinkedBlockingDeque.java:621) > at > org.apache.cassandra.locator.AdaptiveLatencyTracker.add(DynamicEndpointSnitch.java:288) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.receiveTiming(DynamicEndpointSnitch.java:202) > at > org.apache.cassandra.net.MessagingService.addLatency(MessagingService.java:152) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 3 more > ERROR 15:33:48,615 Fatal exception in thread Thread[ReadStage:264,5,main] > java.lang.RuntimeException: java.util.NoSuchElementException > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.util.NoSuchElementException > at > java.util.concurrent.LinkedBlockingDeque.removeFirst(LinkedBlockingDeque.java:401) > at > java.util.concurrent.LinkedBlockingDeque.remove(LinkedBlockingDeque.java:621) > at > org.apache.cassandra.locator.AdaptiveLatencyTracker.add(DynamicEndpointSnitch.java:288) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.receiveTiming(DynamicEndpointSnitch.java:202) > at > org.apache.cassandra.net.MessagingService.addLatency(MessagingService.java:152) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:642) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 3 more > What is happening that AdaptiveLatencyTracker.add is trying to add a latency, > but the deque is full, so it makes a second effort to remove an entry from > the deque and then try to add again. However, when it tries to remove, the > deque has already been emptied by DES.reset call clear() on all the ALTs. > This bug has existed for a long time, but it's very rare and difficult to > trigger. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira