Re: Repair: Issue in netstats
Thanks a lot for clarifying. We'll complete the upgrade of all nodes. Regards. On Mon, Sep 17, 2012 at 3:51 PM, Sylvain Lebresne sylv...@datastax.comwrote: On Mon, Sep 17, 2012 at 11:06 AM, B R software.research.w...@gmail.com wrote: Could this problem be due to running repair on a node upgraded to 1.0.11 but the other node in the cluster is still at 0.8.x ? Yes, repair (as all operation requiring streaming) doesn't work correctly across major Cassandra version. First thing you should do is to finish the upgrade of the nodes. -- Sylvain On Fri, Sep 7, 2012 at 9:11 PM, Sylvain Lebresne sylv...@datastax.com wrote: That obviously shouldn't happen and I don't remember any open ticket related to that. You might want to open a ticket on jira (https://issues.apache.org/jira/browse/CASSANDRA). -- Sylvain On Fri, Sep 7, 2012 at 10:50 AM, B R software.research.w...@gmail.com wrote: We have upgraded a 0.8 cluster to 1.0.11. After upgrading the first node and running upgradesstables, we have run a routine repair operation, This operation has been running for a long time and does not seem to be progressing. Running netstats has shown unexpected values for percentages as shown below. Any clue as to what could be be issue ? bin/nodetool -h 172.16.0.34 netstats Mode: NORMAL Streaming to: /172.16.0.29 /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=116 progress=19946657796608/334406146 - 5964800% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=116 progress=0/179880575 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=12 progress=0/1448134 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=116 progress=0/350403675 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=89 progress=0/27569594 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=1 progress=0/95043 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16617-Data.db sections=1 progress=0/232800 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16612-Data.db sections=1 progress=0/82705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=116 progress=0/724836994 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=116 progress=0/401797714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=2 progress=0/301297 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=3 progress=0/829914 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=2 progress=0/288460 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=13 progress=0/1954639 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=8 progress=0/1187649 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=1 progress=0/141714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=116 progress=0/390168999 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=111 progress=13620592201686/303748754 - 4484163% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=110 progress=0/162808076 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=10 progress=0/1922996 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=111 progress=0/350744309 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=87 progress=0/24364920 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=2 progress=0/228764 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=111 progress=0/720722886 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=111 progress=0/364643588 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=4 progress=0/963207 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=2 progress=0/360024 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=1 progress=0/72842 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=11 progress=0/1381176 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=13 progress=0/3266736 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=2 progress=0/639705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=111 progress=0/358443928 - 0% Nothing streaming from /172.16.0.29 Pool NameActive Pending Completed Commandsn/a 0 19 Responses n/a
Re: Repair: Issue in netstats
Sorry for the delay; been out of the loop. Could this problem be due to running repair on a node upgraded to 1.0.11 but the other node in the cluster is still at 0.8.x ? On Fri, Sep 7, 2012 at 9:11 PM, Sylvain Lebresne sylv...@datastax.comwrote: That obviously shouldn't happen and I don't remember any open ticket related to that. You might want to open a ticket on jira (https://issues.apache.org/jira/browse/CASSANDRA). -- Sylvain On Fri, Sep 7, 2012 at 10:50 AM, B R software.research.w...@gmail.com wrote: We have upgraded a 0.8 cluster to 1.0.11. After upgrading the first node and running upgradesstables, we have run a routine repair operation, This operation has been running for a long time and does not seem to be progressing. Running netstats has shown unexpected values for percentages as shown below. Any clue as to what could be be issue ? bin/nodetool -h 172.16.0.34 netstats Mode: NORMAL Streaming to: /172.16.0.29 /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=116 progress=19946657796608/334406146 - 5964800% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=116 progress=0/179880575 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=12 progress=0/1448134 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=116 progress=0/350403675 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=89 progress=0/27569594 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=1 progress=0/95043 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16617-Data.db sections=1 progress=0/232800 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16612-Data.db sections=1 progress=0/82705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=116 progress=0/724836994 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=116 progress=0/401797714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=2 progress=0/301297 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=3 progress=0/829914 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=2 progress=0/288460 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=13 progress=0/1954639 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=8 progress=0/1187649 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=1 progress=0/141714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=116 progress=0/390168999 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=111 progress=13620592201686/303748754 - 4484163% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=110 progress=0/162808076 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=10 progress=0/1922996 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=111 progress=0/350744309 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=87 progress=0/24364920 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=2 progress=0/228764 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=111 progress=0/720722886 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=111 progress=0/364643588 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=4 progress=0/963207 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=2 progress=0/360024 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=1 progress=0/72842 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=11 progress=0/1381176 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=13 progress=0/3266736 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=2 progress=0/639705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=111 progress=0/358443928 - 0% Nothing streaming from /172.16.0.29 Pool NameActive Pending Completed Commandsn/a 0 19 Responses n/a 02050444 Regards.
Re: Repair: Issue in netstats
On Mon, Sep 17, 2012 at 11:06 AM, B R software.research.w...@gmail.com wrote: Could this problem be due to running repair on a node upgraded to 1.0.11 but the other node in the cluster is still at 0.8.x ? Yes, repair (as all operation requiring streaming) doesn't work correctly across major Cassandra version. First thing you should do is to finish the upgrade of the nodes. -- Sylvain On Fri, Sep 7, 2012 at 9:11 PM, Sylvain Lebresne sylv...@datastax.com wrote: That obviously shouldn't happen and I don't remember any open ticket related to that. You might want to open a ticket on jira (https://issues.apache.org/jira/browse/CASSANDRA). -- Sylvain On Fri, Sep 7, 2012 at 10:50 AM, B R software.research.w...@gmail.com wrote: We have upgraded a 0.8 cluster to 1.0.11. After upgrading the first node and running upgradesstables, we have run a routine repair operation, This operation has been running for a long time and does not seem to be progressing. Running netstats has shown unexpected values for percentages as shown below. Any clue as to what could be be issue ? bin/nodetool -h 172.16.0.34 netstats Mode: NORMAL Streaming to: /172.16.0.29 /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=116 progress=19946657796608/334406146 - 5964800% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=116 progress=0/179880575 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=12 progress=0/1448134 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=116 progress=0/350403675 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=89 progress=0/27569594 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=1 progress=0/95043 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16617-Data.db sections=1 progress=0/232800 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16612-Data.db sections=1 progress=0/82705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=116 progress=0/724836994 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=116 progress=0/401797714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=2 progress=0/301297 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=3 progress=0/829914 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=2 progress=0/288460 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=13 progress=0/1954639 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=8 progress=0/1187649 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=1 progress=0/141714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=116 progress=0/390168999 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=111 progress=13620592201686/303748754 - 4484163% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=110 progress=0/162808076 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=10 progress=0/1922996 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=111 progress=0/350744309 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=87 progress=0/24364920 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=2 progress=0/228764 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=111 progress=0/720722886 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=111 progress=0/364643588 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=4 progress=0/963207 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=2 progress=0/360024 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=1 progress=0/72842 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=11 progress=0/1381176 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=13 progress=0/3266736 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=2 progress=0/639705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=111 progress=0/358443928 - 0% Nothing streaming from /172.16.0.29 Pool NameActive Pending Completed Commandsn/a 0 19 Responses n/a 02050444 Regards.
Repair: Issue in netstats
We have upgraded a 0.8 cluster to 1.0.11. After upgrading the first node and running upgradesstables, we have run a routine repair operation, This operation has been running for a long time and does not seem to be progressing. Running netstats has shown unexpected values for percentages as shown below. Any clue as to what could be be issue ? bin/nodetool -h 172.16.0.34 netstats Mode: NORMAL Streaming to: /172.16.0.29 /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=116 progress=*19946657796608/334406146 - 5964800%* /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=116 progress=0/179880575 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=12 progress=0/1448134 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=116 progress=0/350403675 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=89 progress=0/27569594 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=1 progress=0/95043 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16617-Data.db sections=1 progress=0/232800 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16612-Data.db sections=1 progress=0/82705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=116 progress=0/724836994 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=116 progress=0/401797714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=2 progress=0/301297 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=3 progress=0/829914 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=2 progress=0/288460 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=13 progress=0/1954639 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=8 progress=0/1187649 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=1 progress=0/141714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=116 progress=0/390168999 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=111 progress=*13620592201686/303748754 - 4484163%* /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=110 progress=0/162808076 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=10 progress=0/1922996 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=111 progress=0/350744309 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=87 progress=0/24364920 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=2 progress=0/228764 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=111 progress=0/720722886 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=111 progress=0/364643588 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=4 progress=0/963207 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=2 progress=0/360024 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=1 progress=0/72842 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=11 progress=0/1381176 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=13 progress=0/3266736 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=2 progress=0/639705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=111 progress=0/358443928 - 0% Nothing streaming from /172.16.0.29 Pool NameActive Pending Completed Commandsn/a 0 19 Responses n/a 02050444 Regards.
Re: Repair: Issue in netstats
That obviously shouldn't happen and I don't remember any open ticket related to that. You might want to open a ticket on jira (https://issues.apache.org/jira/browse/CASSANDRA). -- Sylvain On Fri, Sep 7, 2012 at 10:50 AM, B R software.research.w...@gmail.com wrote: We have upgraded a 0.8 cluster to 1.0.11. After upgrading the first node and running upgradesstables, we have run a routine repair operation, This operation has been running for a long time and does not seem to be progressing. Running netstats has shown unexpected values for percentages as shown below. Any clue as to what could be be issue ? bin/nodetool -h 172.16.0.34 netstats Mode: NORMAL Streaming to: /172.16.0.29 /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=116 progress=19946657796608/334406146 - 5964800% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=116 progress=0/179880575 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=12 progress=0/1448134 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=116 progress=0/350403675 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=89 progress=0/27569594 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=1 progress=0/95043 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16617-Data.db sections=1 progress=0/232800 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16612-Data.db sections=1 progress=0/82705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=116 progress=0/724836994 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=116 progress=0/401797714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=2 progress=0/301297 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=3 progress=0/829914 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=2 progress=0/288460 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=13 progress=0/1954639 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=8 progress=0/1187649 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=1 progress=0/141714 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=116 progress=0/390168999 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16609-Data.db sections=111 progress=13620592201686/303748754 - 4484163% /data/cassandra/data/Keyspace1/Standard1-hd-16618-Data.db sections=110 progress=0/162808076 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16620-Data.db sections=10 progress=0/1922996 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16616-Data.db sections=111 progress=0/350744309 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16602-Data.db sections=87 progress=0/24364920 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16615-Data.db sections=2 progress=0/228764 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16603-Data.db sections=111 progress=0/720722886 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16607-Data.db sections=111 progress=0/364643588 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16608-Data.db sections=4 progress=0/963207 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16619-Data.db sections=2 progress=0/360024 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16604-Data.db sections=1 progress=0/72842 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16610-Data.db sections=11 progress=0/1381176 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16606-Data.db sections=13 progress=0/3266736 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16613-Data.db sections=2 progress=0/639705 - 0% /data/cassandra/data/Keyspace1/Standard1-hd-16614-Data.db sections=111 progress=0/358443928 - 0% Nothing streaming from /172.16.0.29 Pool NameActive Pending Completed Commandsn/a 0 19 Responses n/a 02050444 Regards.
