[jira] [Updated] (CASSANDRA-8533) Nodetool Repair hangs on trunk

2015-07-18 Thread Aleksey Yeschenko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-8533:
-
Fix Version/s: (was: 3.x)

 Nodetool Repair hangs on trunk
 --

 Key: CASSANDRA-8533
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8533
 Project: Cassandra
  Issue Type: Bug
Reporter: Philip Thompson
Assignee: Yuki Morishita

 The dtests {{sstableloader_compression_none_to_snappy_test}} and 
 {{sstableloader_compression_none_to_deflate_test}} in  
 {{sstable_generation_loading_test.py}} are hanging on trunk during repair. 
 Test output can be seen here: 
 http://cassci.datastax.com/job/trunk_dtest/752/console
 Here is the jstack output for nodetool
 {code}
 2014-12-23 12:11:46
 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):
 Attach Listener daemon prio=5 tid=0x7fe51301e800 nid=0x370b waiting on 
 condition [0x]
java.lang.Thread.State: RUNNABLE
 ClientNotifForwarder-1 daemon prio=5 tid=0x7fe514969800 nid=0x5d03 
 runnable [0x000113395000]
java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:152)
   at java.net.SocketInputStream.read(SocketInputStream.java:122)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
   - locked 0x0007fd504398 (a java.io.BufferedInputStream)
   at java.io.DataInputStream.readByte(DataInputStream.java:265)
   at 
 sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:214)
   at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
   at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
   at 
 javax.management.remote.rmi.RMIConnectionImpl_Stub.fetchNotifications(Unknown 
 Source)
   at 
 javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1342)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107)
 JMX client heartbeat 2 daemon prio=5 tid=0x7fe5148f8000 nid=0x5b03 
 waiting on condition [0x000113292000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at 
 com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:174)
   at java.lang.Thread.run(Thread.java:745)
 GC Daemon daemon prio=5 tid=0x7fe5148db800 nid=0x5903 in Object.wait() 
 [0x00011318f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x0007f8009570 (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:117)
   - locked 0x0007f8009570 (a sun.misc.GC$LatencyLock)
 RMI RenewClean-[10.150.0.64:7100] daemon prio=5 tid=0x7fe5148d3000 
 nid=0x5703 in Object.wait() [0x00011308c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x0007f8010ae0 (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   - locked 0x0007f8010ae0 (a java.lang.ref.ReferenceQueue$Lock)
   at 
 sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535)
   at java.lang.Thread.run(Thread.java:745)
 RMI Scheduler(0) daemon prio=5 tid=0x7fe51490c800 nid=0x5607 waiting on 
 condition [0x000112f89000]
java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  0x0007f8018ad0 (a 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   at 
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   at 
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   at 
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   at

[jira] [Updated] (CASSANDRA-8533) Nodetool Repair hangs on trunk

2015-01-06 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-8533:
--
Assignee: Yuki Morishita

 Nodetool Repair hangs on trunk
 --

 Key: CASSANDRA-8533
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8533
 Project: Cassandra
  Issue Type: Bug
Reporter: Philip Thompson
Assignee: Yuki Morishita
 Fix For: 3.0


 The dtests {{sstableloader_compression_none_to_snappy_test}} and 
 {{sstableloader_compression_none_to_deflate_test}} in  
 {{sstable_generation_loading_test.py}} are hanging on trunk during repair. 
 Test output can be seen here: 
 http://cassci.datastax.com/job/trunk_dtest/752/console
 Here is the jstack output for nodetool
 {code}
 2014-12-23 12:11:46
 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):
 Attach Listener daemon prio=5 tid=0x7fe51301e800 nid=0x370b waiting on 
 condition [0x]
java.lang.Thread.State: RUNNABLE
 ClientNotifForwarder-1 daemon prio=5 tid=0x7fe514969800 nid=0x5d03 
 runnable [0x000113395000]
java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:152)
   at java.net.SocketInputStream.read(SocketInputStream.java:122)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
   - locked 0x0007fd504398 (a java.io.BufferedInputStream)
   at java.io.DataInputStream.readByte(DataInputStream.java:265)
   at 
 sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:214)
   at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
   at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
   at 
 javax.management.remote.rmi.RMIConnectionImpl_Stub.fetchNotifications(Unknown 
 Source)
   at 
 javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1342)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107)
 JMX client heartbeat 2 daemon prio=5 tid=0x7fe5148f8000 nid=0x5b03 
 waiting on condition [0x000113292000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at 
 com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:174)
   at java.lang.Thread.run(Thread.java:745)
 GC Daemon daemon prio=5 tid=0x7fe5148db800 nid=0x5903 in Object.wait() 
 [0x00011318f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x0007f8009570 (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:117)
   - locked 0x0007f8009570 (a sun.misc.GC$LatencyLock)
 RMI RenewClean-[10.150.0.64:7100] daemon prio=5 tid=0x7fe5148d3000 
 nid=0x5703 in Object.wait() [0x00011308c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x0007f8010ae0 (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   - locked 0x0007f8010ae0 (a java.lang.ref.ReferenceQueue$Lock)
   at 
 sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535)
   at java.lang.Thread.run(Thread.java:745)
 RMI Scheduler(0) daemon prio=5 tid=0x7fe51490c800 nid=0x5607 waiting on 
 condition [0x000112f89000]
java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  0x0007f8018ad0 (a 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   at 
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   at 
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   at 
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   at

[jira] [Updated] (CASSANDRA-8533) Nodetool Repair hangs on trunk

2014-12-23 Thread Philip Thompson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-8533:
---
Summary: Nodetool Repair hangs on trunk  (was: Repair hangs on trunk)

 Nodetool Repair hangs on trunk
 --

 Key: CASSANDRA-8533
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8533
 Project: Cassandra
  Issue Type: Bug
Reporter: Philip Thompson
 Fix For: 3.0


 The dtests {{sstableloader_compression_none_to_snappy_test}} and 
 {{sstableloader_compression_none_to_deflate_test}} in  
 {{sstable_generation_loading_test.py}} are hanging on trunk during repair. 
 Test output can be seen here: 
 http://cassci.datastax.com/job/trunk_dtest/752/console
 Here is the jstack output for nodetool
 {code}
 2014-12-23 12:11:46
 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):
 Attach Listener daemon prio=5 tid=0x7fe51301e800 nid=0x370b waiting on 
 condition [0x]
java.lang.Thread.State: RUNNABLE
 ClientNotifForwarder-1 daemon prio=5 tid=0x7fe514969800 nid=0x5d03 
 runnable [0x000113395000]
java.lang.Thread.State: RUNNABLE
   at java.net.SocketInputStream.socketRead0(Native Method)
   at java.net.SocketInputStream.read(SocketInputStream.java:152)
   at java.net.SocketInputStream.read(SocketInputStream.java:122)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
   - locked 0x0007fd504398 (a java.io.BufferedInputStream)
   at java.io.DataInputStream.readByte(DataInputStream.java:265)
   at 
 sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:214)
   at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
   at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
   at 
 javax.management.remote.rmi.RMIConnectionImpl_Stub.fetchNotifications(Unknown 
 Source)
   at 
 javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1342)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:587)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:470)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:451)
   at 
 com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:107)
 JMX client heartbeat 2 daemon prio=5 tid=0x7fe5148f8000 nid=0x5b03 
 waiting on condition [0x000113292000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
   at java.lang.Thread.sleep(Native Method)
   at 
 com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:174)
   at java.lang.Thread.run(Thread.java:745)
 GC Daemon daemon prio=5 tid=0x7fe5148db800 nid=0x5903 in Object.wait() 
 [0x00011318f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x0007f8009570 (a sun.misc.GC$LatencyLock)
   at sun.misc.GC$Daemon.run(GC.java:117)
   - locked 0x0007f8009570 (a sun.misc.GC$LatencyLock)
 RMI RenewClean-[10.150.0.64:7100] daemon prio=5 tid=0x7fe5148d3000 
 nid=0x5703 in Object.wait() [0x00011308c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x0007f8010ae0 (a java.lang.ref.ReferenceQueue$Lock)
   at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   - locked 0x0007f8010ae0 (a java.lang.ref.ReferenceQueue$Lock)
   at 
 sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535)
   at java.lang.Thread.run(Thread.java:745)
 RMI Scheduler(0) daemon prio=5 tid=0x7fe51490c800 nid=0x5607 waiting on 
 condition [0x000112f89000]
java.lang.Thread.State: WAITING (parking)
   at sun.misc.Unsafe.park(Native Method)
   - parking to wait for  0x0007f8018ad0 (a 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   at 
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   at 
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   at 
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   at