Victor Trac created CASSANDRA-11981: ---------------------------------------
Summary: Cassandra 2.2 -> 3.5 upgradesstables results in error: null Key: CASSANDRA-11981 URL: https://issues.apache.org/jira/browse/CASSANDRA-11981 Project: Cassandra Issue Type: Bug Environment: # java -version openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) # uname -a Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux # rpm -qa datastax-ddc datastax-ddc-3.5.0-1.noarch Reporter: Victor Trac We upgraded from Cassandra 2.2.1 -> 3.5 on a 6-node cluster. I ran 'nodetool upgradesstables' on all nodes successfully except for one. It would take a 2-3 days ('nodetool netstats` showed continual progress), but then the server would error out with a java.io.EOFException. After trying upgradesstables several times, I tried a scrub. Again, it ran for a couple of days and then errored out with a very similar error: {code} # nodetool scrub error: null -- StackTrace -- java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:267) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:215) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) at com.sun.proxy.$Proxy7.scrub(Unknown Source) at org.apache.cassandra.tools.NodeProbe.scrub(NodeProbe.java:248) at org.apache.cassandra.tools.NodeProbe.scrub(NodeProbe.java:280) at org.apache.cassandra.tools.nodetool.Scrub.execute(Scrub.java:66) at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:250) at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:164) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)