Re: sstablescrum fails with OOM

[Jeff Jirsa]

This is not guaranteed to be safe

If the corrupted sstable has a tombstone past gc grace, and another sstable has 
shadowed deleted data, removing the corrupt sstable will cause the data to come 
back to life, and repair will spread it around the ring

If that’s problematic to you, you should consider the entire node failed, run 
repair among the surviving replicas and then replace the down server

If you don’t do deletes, and write with consistency higher than ONE, there’s a 
bit less risk to removing a single sstable


-- 
Jeff Jirsa


> On Nov 2, 2017, at 7:58 PM, sai krishnam raju potturi  
> wrote:
> 
> Yes. Move the corrupt sstable, and run a repair on this node, so that it gets 
> in sync with it's peers.
> 
>> On Thu, Nov 2, 2017 at 6:12 PM, Shashi Yachavaram  
>> wrote:
>> We are cassandra 2.0.17 and have corrupted sstables. Ran offline 
>> sstablescrub but it fails with OOM. Increased the MAX_HEAP_SIZE to 8G it 
>> still fails. 
>> 
>> Can we move the corrupted sstable file and rerun sstablescrub followed by 
>> repair.
>> 
>> -shashi..
> 

cassandra gc issue

[Peng Xiao]

All,


We noticed that the response time jumps very high sometime. The following is 
from the cassandra gc log.


   [Eden: 760.0M(760.0M)->0.0B(11.2G) Survivors: 264.0M->96.0M Heap: 
7657.7M(20.0G)->6893.3M(20.0G)]
Heap after GC invocations=43481 (full 0):
 garbage-first heap   total 20971520K, used 7058765K [0x0002c000, 
0x0002c0805000, 0x0007c000)
  region size 8192K, 12 young (98304K), 12 survivors (98304K)
 Metaspace   used 37426K, capacity 37810K, committed 38144K, reserved 
1083392K
  class spaceused 3930K, capacity 4025K, committed 4096K, reserved 1048576K



Could anyone please advise?


Thanks,
Peng Xiao

Re: sstablescrum fails with OOM

[sai krishnam raju potturi]

Yes. Move the corrupt sstable, and run a repair on this node, so that it
gets in sync with it's peers.

On Thu, Nov 2, 2017 at 6:12 PM, Shashi Yachavaram 
wrote:

> We are cassandra 2.0.17 and have corrupted sstables. Ran offline
> sstablescrub but it fails with OOM. Increased the MAX_HEAP_SIZE to 8G it
> still fails.
>
> Can we move the corrupted sstable file and rerun sstablescrub followed by
> repair.
>
> -shashi..
>

Cassandra using a ton of native memory

[Austin Sharp]

Hi,



I have a problem with Cassandra 3.11.0 on Windows. I'm testing a workload w= 
ith a lot of read-then-writes that had no significant problems on Cassandra=  
2.x. However, now when this workload continues for a while (perhaps an hou= r), 
Cassandra or its JVM effectively use up all of the machine's 16GB of me= mory. 
Cassandra is started with -Xmx2147M, and JMX shows <2GB heap memory a= nd 
<100MB of off-heap memory. However, when I use something like Process Ex= 
plorer, I see that Cassandra has 10 to 11GB of memory in its working set, a= nd 
Windows shows essentially no free memory at all. Once the system has no = free 
memory, other processes suffer long sequences of unresponsiveness.



I can't see anything terribly wrong from JMX metrics or log files - they ne= 
ver show more than 1GB of non-heap memory. Where should I look to investiga= te 
this further?



Thanks,

Austin

sstablescrum fails with OOM

[Shashi Yachavaram]

We are cassandra 2.0.17 and have corrupted sstables. Ran offline
sstablescrub but it fails with OOM. Increased the MAX_HEAP_SIZE to 8G it
still fails.

Can we move the corrupted sstable file and rerun sstablescrub followed by
repair.

-shashi..

Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version

[kurt greaves]

Well, pretty sure they still are. at least the mutation one is. but you
should really use the dedicated metrics for this.

On 3 Nov. 2017 01:38, "Anumod Mullachery" 
wrote:

> thanks ..
>
> so the dropped hints & messages are not captured in cassandra logs, post
> 3.x vs 2.x.
>
> -Anumod
>
>
>
> On Wed, Nov 1, 2017 at 4:50 PM, kurt greaves  wrote:
>
>> You can get dropped message statistics over JMX. for example nodetool
>> tpstats has a counter for dropped hints from startup. that would be the
>> preferred method for tracking this info, rather than parsing logs
>>
>> On 2 Nov. 2017 6:24 am, "Anumod Mullachery" 
>> wrote:
>>
>>
>> Hi All,
>>
>> In cassandra  v 2.1.15 , I'm able to pull the hints drop and dropped
>> messages from cassandra.log as below-
>>
>>
>> dropped hints-->
>>
>> "/opt/xcal/apps/cassandra/logs/cassandra.log
>> 
>> --> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints,
>> because node is down past configured hint window."
>>
>> Dropped messages -->
>>
>> "/opt/xcal/apps/cassandra/logs/cassandra.log
>> 
>> --> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"
>>
>>
>> But in Cassandra *V 3.10* , I'm not able to get the cassandra logs as v
>> 2.1.15.
>>
>> I'm not able to get any info from the cassandra logs for dropped hints &
>> dropped messages. ( I was testing this on Test cluster by stopping the
>> cassandra services on one of the node , where the hints window is set to 05
>> Mins and kept the node down for 1 hour , and monitoring the cassandra logs,
>> but nothing generated on dropped hints on any other nodes in the cluster ).
>>
>> The only message I found is on the down node is that - "Paused hints
>> dispatch".
>>
>> Can some one put some light on this issue on 3.10 , is there any changes
>> in the hints & dropped messages , logs or the process.
>>
>> thanks in advance,
>>
>> regards,
>>
>> Anumod.
>>
>>
>>
>

What is OneMinuteRate in Write Latency?

[AI Rumman]

Hi,

I am trying to calculate the Read/second and Write/Second in my Cassandra
2.1 cluster. After searching and reading, I came to know about JMX bean
"org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency".
Here I can see oneMinuteRate. I have started a brand new cluster and
started collected these metrics from 0.
When I started my first record, I can see

Count = 1
> OneMinuteRate = 0.01599111...


Does it mean that my write/s is 0.0159911? Or does it mean that based on 1
minute data, my write latency is 0.01599 where Write Latency refers to the
response time for writing a record?

Please help me understand the value.

Thanks.

Re: Cassandra 3.10 - Hints & Dropped messages logs Vs Cass 2.x version

[Anumod Mullachery]

thanks ..

so the dropped hints & messages are not captured in cassandra logs, post
3.x vs 2.x.

-Anumod



On Wed, Nov 1, 2017 at 4:50 PM, kurt greaves  wrote:

> You can get dropped message statistics over JMX. for example nodetool
> tpstats has a counter for dropped hints from startup. that would be the
> preferred method for tracking this info, rather than parsing logs
>
> On 2 Nov. 2017 6:24 am, "Anumod Mullachery" 
> wrote:
>
>
> Hi All,
>
> In cassandra  v 2.1.15 , I'm able to pull the hints drop and dropped
> messages from cassandra.log as below-
>
>
> dropped hints-->
>
> "/opt/xcal/apps/cassandra/logs/cassandra.log
> 
> --> HintedHandoffMetrics.java:79 - /96.115.91.69 has 5 dropped hints,
> because node is down past configured hint window."
>
> Dropped messages -->
>
> "/opt/xcal/apps/cassandra/logs/cassandra.log
> 
> --> NFO MessagingService 1 MUTATION messages dropped in last 5000ms"
>
>
> But in Cassandra *V 3.10* , I'm not able to get the cassandra logs as v
> 2.1.15.
>
> I'm not able to get any info from the cassandra logs for dropped hints &
> dropped messages. ( I was testing this on Test cluster by stopping the
> cassandra services on one of the node , where the hints window is set to 05
> Mins and kept the node down for 1 hour , and monitoring the cassandra logs,
> but nothing generated on dropped hints on any other nodes in the cluster ).
>
> The only message I found is on the down node is that - "Paused hints
> dispatch".
>
> Can some one put some light on this issue on 3.10 , is there any changes
> in the hints & dropped messages , logs or the process.
>
> thanks in advance,
>
> regards,
>
> Anumod.
>
>
>

Re: Why Cassandra need full repair after incremental repair

[Blake Eggleston]

Because in theory, corruption of your repaired dataset is possible, which 
incremental repair won’t fix. 

In practice pre-4.0 incremental repair has some flaws that can bring deleted 
data back to life in some cases, which this would address. 

You should also evaluate whether pre-4.0 incremental repair is saving you time. 
The same flaws can cause *a lot* of over streaming, which may negate the 
benefit of repairing only the unrepaired data.

> On Nov 2, 2017, at 2:17 AM, dayu  wrote:
> 
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesWhen.html
> 
> So you means i am misleading by this statements. The full repair only needed 
> when node failure + replacement, or adding a datacenter. right?
> 
> 
> 
> 
> At 2017-11-02 15:54:49, "kurt greaves"  wrote:
> Where are you seeing this? If your incremental repairs work properly, full 
> repair is only needed in certain situations, like after node failure + 
> replacement, or adding a datacenter.
> 
> 
>  

Re: Cqlsh unable to switch keyspace after Cassandra upgrade.

[Blake Eggleston]

Looks like a bug, could you open a jira?

> On Nov 2, 2017, at 2:08 AM, Mikhail Tsaplin  wrote:
> 
> Hi,
> I've upgraded Cassandra from 2.1.6 to 3.0.9 on three nodes cluster. After 
> upgrade 
> cqlsh shows following error when trying to run "use {keyspace};" command:
> 'ResponseFuture' object has no attribute 'is_schema_agreed'
> 
> Actual upgrade was done on Ubuntu 16.04 by running "apt-get upgrade 
> cassandra" command.
> Apt repository is deb http://debian.datastax.com/community stable main.
> Following parameters were migrated from former cassandra.yaml:
> cluster_name, num_tokens, data_file_directories, commit_log_directory, 
> saved_caches_directory, seeds, listen_address, rpc_address, initial_token, 
> auto_bootstrap.
> 
> Later I did additional test - fetched 3.0.15 binary distribution from 
> cassandra.apache.org and tried to run cassandra from this distr - same error:
> $ ./bin/cqlsh
> Connected to cellwize.cassandra at 172.31.17.42:9042.
> [cqlsh 5.0.1 | Cassandra 3.0.15 | CQL spec 3.4.0 | Native protocol v4]
> Use HELP for help.
> cqlsh> use listener ;
> 'ResponseFuture' object has no attribute 'is_schema_agreed'
> cqlsh> 
> 
> What could be the reason?

During a rolling restart, one node marked as "shutdown" indefinitely by all other nodes

[Yap Sok Ann]

We have a large cluster running 2.1.19, with 3 datacenters:
- xxx: 220 nodes
- yyy: 220 nodes
- zzz: 500 nodes

The ping time between the datacenters are:
- xxx to yyy:  50 ms
- xxx to zzz: 240 ms
- yyy to zzz: 200 ms

There are some added complications such that:
- our team is managing xxx and yyy only, and have no ssh/logging access
to zzz.
- the link to zzz can be slow/unstable at times.

In a maintenance window a few days ago, we did a rolling restart,
bouncing one node at a time in both xxx and yyy concurrently, with
command `nodetool drain && sudo service cassandra restart && sleep 30`.

In the process, the node xxx-167 somehow was marked as "shutdown" by all
other nodes, and remained as such indefinitely.

We only noticed that about 8 hours later. Then, we checked the
gossipinfo for xxx-167:

- from the node itself:

xxx-167.domain/1.2.3.4
  generation:1509301942
  heartbeat:47368
  STATUS:14:NORMAL,
  LOAD:47293:
  SCHEMA:10:
  DC:29:xxx
  RACK:31:a
  RELEASE_VERSION:4:2.1.19
  RPC_ADDRESS:3:1.2.3.4
  SEVERITY:47367:0.0
  NET_VERSION:1:8
  HOST_ID:2:
  TOKENS:13:

- from yyy-001:

/1.2.3.4
  generation:1509301942
  heartbeat:2147483647
  STATUS:608083:shutdown,true
  LOAD:22:
  SCHEMA:10:
  DC:29:xxx
  RACK:31:a
  RELEASE_VERSION:4:2.1.19
  RPC_ADDRESS:3:1.2.3.4
  SEVERITY:176:0.0
  NET_VERSION:1:8
  HOST_ID:2:
  TOKENS:13:

Another 2 hours later, we decided to just restart cassandra on xxx-167.
That fixed the problem.

Here's some logs from nodes in xxx and yyy, that could be relevant to
the incident:

2017-10-29 18:31:55,324 [yyy-081] StorageService.java:1138 - DRAINING:
starting drain process
2017-10-29 18:31:57,897 [yyy-081] StorageService.java:1138 - DRAINED
2017-10-29 18:32:06,116 [xxx-167] StorageService.java:1138 - DRAINING:
starting drain process
2017-10-29 18:32:06,357 [group-a] Gossiper.java:1010 - InetAddress
xxx-167 is now DOWN
... repeat ...
2017-10-29 18:32:06,552 [group-a] Gossiper.java:1010 - InetAddress
xxx-167 is now DOWN
2017-10-29 18:32:08,534 [xxx-167] StorageService.java:1138 - DRAINED
2017-10-29 18:32:10,337 [yyy-081] CassandraDaemon.java:155 - Hostname:
yyy-081.domain
2017-10-29 18:32:12,939 [yyy-081] StorageService.java:807 - Starting up
server gossip
2017-10-29 18:32:13,484 [yyy-081] OutboundTcpConnection.java:496 -
Handshaking version with yyy-173
2017-10-29 18:32:13,486 [yyy-173] OutboundTcpConnection.java:496 -
Handshaking version with yyy-081
2017-10-29 18:32:13,493 [yyy-081] Gossiper.java:1029 - Node xxx-167 has
restarted, now UP
2017-10-29 18:32:13,495 [yyy-081] StorageService.java:1715 - Node
xxx-167 state jump to shutdown
2017-10-29 18:32:13,499 [yyy-081] Gossiper.java:1010 - InetAddress
xxx-167 is now DOWN
2017-10-29 18:32:20,182 [xxx-167] CassandraDaemon.java:155 - Hostname:
xxx-167.domain
2017-10-29 18:32:22,304 [xxx-167] StorageService.java:807 - Starting up
server gossip
2017-10-29 18:32:22,463 [yyy-081] OutboundTcpConnection.java:496 -
Handshaking version with xxx-167
2017-10-29 18:32:22,780 [xxx-167] OutboundTcpConnection.java:496 -
Handshaking version with yyy-173
2017-10-29 18:32:23,857 [xxx-167] OutboundTcpConnection.java:496 -
Handshaking version with yyy-081
2017-10-29 18:32:25,797 [yyy-173] Gossiper.java:1029 - Node xxx-167 has
restarted, now UP
2017-10-29 18:32:25,847 [yyy-173] OutboundTcpConnection.java:496 -
Handshaking version with xxx-167
2017-10-29 18:32:27,113 [yyy-081] Gossiper.java:1029 - Node xxx-167 has
restarted, now UP
2017-10-29 18:33:13,443 [yyy-173] Gossiper.java:1010 - InetAddress
xxx-167 is now DOWN
2017-10-29 18:33:13,628 [group-b] StorageService.java:1715 - Node
xxx-167 state jump to shutdown
... repeat ...
2017-10-29 18:33:28,809 [group-b] StorageService.java:1715 - Node
xxx-167 state jump to shutdown

Notes:
- "group-a" includes 437 nodes in xxx and yyy, except:
  - xxx-167: the draining node
  - yyy-081: still starting up
  - yyy-173: somehow didn't notice that xxx-167 was down

- "group-b" includes 438 nodes in xxx and yyy, except:
  - xxx-167: the wrongly marked as shutdown node
  - yyy-173: "patient zero" that suddenly thought that xxx-167 was down

Obviously, we would like to know how can this happen, and if there is
any way to prevent it from happening again. Some other questions:

1. Should I create a jira issue?

2. For rolling restart, is it better to do so one datacenter at a time?

3. For graceful shutdown, is the command `nodetool drain && sudo service
cassandra restart` sufficient? The internet suggests to do `nodetool
disablebinary && nodetool disablethrift && nodetool disablegossip`
before `nodetool drain`. However, looking at the 2.1 source code,
`nodetool drain` already does the same things before draining.

Thanks

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re:Re: Why Cassandra need full repair after incremental repair

[dayu]

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesWhen.html
So you means i am misleading by this statements. The full repair only needed 
when node failure + replacement, or adding a datacenter. right?





At 2017-11-02 15:54:49, "kurt greaves"  wrote:

Where are you seeing this? If your incremental repairs work properly, full 
repair is only needed in certain situations, like after node failure + 
replacement, or adding a datacenter.

Cqlsh unable to switch keyspace after Cassandra upgrade.

[Mikhail Tsaplin]

Hi,
I've upgraded Cassandra from 2.1.6 to 3.0.9 on three nodes cluster. After
upgrade
cqlsh shows following error when trying to run "use {keyspace};" command:
'ResponseFuture' object has no attribute 'is_schema_agreed'

Actual upgrade was done on Ubuntu 16.04 by running "apt-get upgrade
cassandra" command.
Apt repository is deb http://debian.datastax.com/community stable main.
Following parameters were migrated from former cassandra.yaml:
cluster_name, num_tokens, data_file_directories, commit_log_directory,
saved_caches_directory, seeds, listen_address, rpc_address, initial_token,
auto_bootstrap.

Later I did additional test - fetched 3.0.15 binary distribution from
cassandra.apache.org and tried to run cassandra from this distr - same
error:
$ ./bin/cqlsh
Connected to cellwize.cassandra at 172.31.17.42:9042.
[cqlsh 5.0.1 | Cassandra 3.0.15 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cqlsh> use listener ;
'ResponseFuture' object has no attribute 'is_schema_agreed'
cqlsh>

What could be the reason?

Re: Why Cassandra need full repair after incremental repair

[kurt greaves]

Where are you seeing this? If your incremental repairs work properly, full
repair is only needed in certain situations, like after node failure +
replacement, or adding a datacenter.​