Re: Cluster is unbalanced

[anil patimidi]

what is your keyspace configuration. Do you have all the keyspaces
configured for both DCs?

can you run below query from cqlsh and see if the keyspace is configured to
use both DCs

select * from system.schema_keyspaces;   # if your cluster is on 2.1 or less
select * from system_schema.keyspaces# for 3.0 clusters

- Anil


On Mon, Jun 18, 2018 at 11:06 AM, learner dba <
cassandra...@yahoo.com.invalid> wrote:

> Hi,
>
> Data volume varies a lot in our two DC cluster:
>
>  Load   Tokens   Owns
>
>  20.01 GiB  256  ?
>
>  65.32 GiB  256  ?
>
>  60.09 GiB  256  ?
>
>  46.95 GiB  256  ?
>
>  50.73 GiB  256  ?
>
> kaiprodv2
>
> =
>
> /Leaving/Joining/Moving
>
>  Load   Tokens   Owns
>
>  25.19 GiB  256  ?
>
>  30.26 GiB  256  ?
>
>  9.82 GiB   256  ?
>
>  20.54 GiB  256  ?
>
>  9.7 GiB256  ?
>
> I ran clearsnapshot, garbagecollect and cleanup, but it increased the size
> on heavier nodes instead of decreasing. Based on nodetool cfstats, I can
> see partition keys on each node varies a lot:
>
> Number of partitions (estimate): 3142552
>
> Number of partitions (estimate): 15625442
>
> Number of partitions (estimate): 15244021
>
> Number of partitions (estimate): 9592992
> Number of partitions (estimate): 15839280
>
> How can I diagnose this imbalance further?
>
>

Re: 3.11.2 memory leak

[kurt greaves]

At this point I'd wait for 3.11.3. If you can't, you can get away with
backporting a few repair fixes or just doing sub range repairs on 3.11.2

On Wed., 20 Jun. 2018, 01:10 Abdul Patel,  wrote:

> Hi All,
>
> Do we kmow whats the stable version for now if u wish to upgrade ?
>
> On Tuesday, June 5, 2018, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
>> Jeff,
>>
>>
>>
>> FWIW, when talking about
>> https://issues.apache.org/jira/browse/CASSANDRA-13929, there is a patch
>> available since March without getting further attention.
>>
>>
>>
>> Regards,
>>
>> Thomas
>>
>>
>>
>> *From:* Jeff Jirsa [mailto:jji...@gmail.com]
>> *Sent:* Dienstag, 05. Juni 2018 00:51
>> *To:* cassandra 
>> *Subject:* Re: 3.11.2 memory leak
>>
>>
>>
>> There have been a few people who have reported it, but nobody (yet) has
>> offered a patch to fix it. It would be good to have a reliable way to
>> repro, and/or an analysis of a heap dump demonstrating the problem (what's
>> actually retained at the time you're OOM'ing).
>>
>>
>>
>> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel  wrote:
>>
>> Hi All,
>>
>>
>>
>> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
>>
>> It was working fine for a 1.5 weeks then suddenly nodetool info startee
>> reporting 80% and more memory consumption.
>>
>> Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4
>> nodes of cluster-single DC.
>>
>> Now after 8 days i again see 80% + usage and its 16gb and above ..which
>> we never saw before .
>>
>> Seems like memory leak bug?
>>
>> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
>> because of this.
>>
>> If not 3.11.2 whats the next best stable release we have now?
>>
>>
>> The contents of this e-mail are intended for the named addressee only. It
>> contains information that may be confidential. Unless you are the named
>> addressee or an authorized designee, you may not copy or use it, or
>> disclose it to anyone else. If you received it in error please notify us
>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>> number FN 91482h) is a company registered in Linz whose registered office
>> is at 4040 Linz, Austria, Freistädterstraße 313
>>
>

Re: RE: RE: [EXTERNAL] Cluster is unbalanced

[Joshua Galbraith]

> id text PRIMARY KEY

What values are written to this id field? Can you give us some examples or
explain the general use case?

On Tue, Jun 19, 2018 at 1:18 PM, learner dba  wrote:

> Hi Sean,
>
> Here is create table:
>
> CREATE TABLE ks.cf (
>
> id text PRIMARY KEY,
>
> accessdata blob
>
> ) WITH bloom_filter_fp_chance = 0.01
>
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
> AND comment = ''
>
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>
> AND crc_check_chance = 1.0
>
> AND dclocal_read_repair_chance = 0.1
>
> AND default_time_to_live = 0
>
> AND gc_grace_seconds = 864000
>
> AND max_index_interval = 2048
>
> AND memtable_flush_period_in_ms = 0
>
> AND min_index_interval = 128
>
> AND read_repair_chance = 0.0
>
> AND speculative_retry = '99PERCENTILE';
> Nodetool status:
>
> Datacenter: dc1
>
> ===
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> --  Address Load   Tokens   Owns (effective)  Host ID
>   Rack
>
> UN  x   20.66 GiB  256  61.4%
> f4f54949-83c9-419b-9a43-cb630b36d8c2  RAC1
>
> UN  x  65.77 GiB  256  59.3%
> 3db430ae-45ef-4746-a273-bc1f66ac8981  RAC1
>
> UN  xx  60.58 GiB  256  58.4%
> 1f23e869-1823-4b75-8d3e-f9b32acba9a6  RAC1
>
> UN  x  47.08 GiB  256  57.5%
> 7aca9a36-823f-4185-be44-c1464a799084  RAC1
>
> UN  x  51.47 GiB  256  63.4%
> 18cff010-9b83-4cf8-9dc2-f05ac63df402  RAC1
>
> Datacenter: dc2
>
> 
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> --  Address Load   Tokens   Owns (effective)  Host ID
>   Rack
>
> UN     24.37 GiB  256  59.5%
> 1b694180-210a-4b75-8f2a-748f4a5b6a3d  RAC1
>
> UN  x 30.76 GiB  256  56.7%
> 597bac04-c57a-4487-8924-72e171e45514  RAC1
>
> UN    10.73 GiB  256  63.9%
> 6e7e474e-e292-4433-afd4-372d30e0f3e1  RAC1
>
> UN  xx 19.77 GiB  256  61.5%
> 58751418-7b76-40f7-8b8f-a5bf8fe7d9a2  RAC1
>
> UN  x  10.33 GiB  256  58.4%
> 6d58d006-2095-449c-8c67-50e8cbdfe7a7  RAC1
>
>
> cassandra-rackdc.properties:
>
> dc=dc1
> rack=RAC1 --> same in all nodes
>
> cassandra.yaml:
> num_tokens: 256
>
> endpoint_snitch: GossipingPropertyFileSnitch
> I can see cassandra-topology.properties, I believe it shouldn't be there
> with GossipPropertyFileSnitch. Can this file be causing any trouble in data
> distribution.
>
> cat /opt/cassandra/conf/cassandra-topology.properties
>
> # Licensed to the Apache Software Foundation (ASF) under one
>
> # or more contributor license agreements.  See the NOTICE file
>
> # distributed with this work for additional information
>
> # regarding copyright ownership.  The ASF licenses this file
>
> # to you under the Apache License, Version 2.0 (the
>
> # "License"); you may not use this file except in compliance
>
> # with the License.  You may obtain a copy of the License at
>
> #
>
> # http://www.apache.org/licenses/LICENSE-2.0
>
> #
>
> # Unless required by applicable law or agreed to in writing, software
>
> # distributed under the License is distributed on an "AS IS" BASIS,
>
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>
> # See the License for the specific language governing permissions and
>
> # limitations under the License.
>
>
> # Cassandra Node IP=Data Center:Rack
>
> 192.168.1.100=DC1:RAC1
>
> 192.168.2.200=DC2:RAC2
>
>
> 10.0.0.10=DC1:RAC1
>
> 10.0.0.11=DC1:RAC1
>
> 10.0.0.12=DC1:RAC2
>
>
> 10.20.114.10=DC2:RAC1
>
> 10.20.114.11=DC2:RAC1
>
>
> 10.21.119.13=DC3:RAC1
>
> 10.21.119.10=DC3:RAC1
>
>
> 10.0.0.13=DC1:RAC2
>
> 10.21.119.14=DC3:RAC2
>
> 10.20.114.15=DC2:RAC2
>
>
> # default for unknown nodes
>
> default=DC1:r1
>
>
> # Native IPv6 is supported, however you must escape the colon in the IPv6
> Address
>
> # Also be sure to comment out JVM_OPTS="$JVM_OPTS
> -Djava.net.preferIPv4Stack=true"
>
> # in cassandra-env.sh
>
> fe80\:0\:0\:0\:202\:b3ff\:fe1e\:8329=DC1:RAC3
>
>
>
>
> On Tuesday, June 19, 2018, 12:51:34 PM EDT, Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
>
> You are correct that the cluster decides where data goes (based on the
> hash of the partition key). However, if you choose a “bad” partition key,
> you may not get good distribution of the data, because the hash is
> deterministic (it always goes to the same nodes/replicas). For example, if
> you have a partition key of a datetime, it is possible that there is more
> data written for a certain time period – thus a larger partition and an
> imbalance across the cluster. Choosing a “good” partition key is one of the
> most important decisions for a Cassandra table.
>
>
>
> Also, I 

how to avoid lightwieght transactions

[manuj singh]

Hi all,
we have a use case where we need to update frequently our rows. Now in
order to do so and so that we dont override updates we have to resort to
lightweight transactions.
Since lightweight is expensive(could be 4 times as expensive as normal
insert) , how do we model around it.

e.g i have a table where

CREATE TABLE multirow (

id text,

time text,

transcation_type text,

status text,

PRIMARY KEY (id, time)

)


So lets say we update status column multiple times. So first time we update
we also have to make sure that the transaction exists otherwise normal
update will insert it and then the original insert comes in and it will
override the update.

So in order to fix that we need to use light weight transactions.


Is there another way i can model this so that we can avoid the lightweight
transactions.



Thanks

Re: sstableloader from dse 4.8.4 to apache cassandra 3.11.1

[rajpal reddy]

Never mind found it. its not a supported version.

> On Jun 19, 2018, at 2:41 PM, rajpal reddy  wrote:
> 
> 
> Hello,
> 
> I’m trying to use sstablloader from dse 4.8.4( 2.1.12) to apache 3.11.1, i’m 
> getting below error. but works fine when i use stableloader dse 5.1.2(apache 
> 3.11.0)
> Could not retrieve endpoint ranges: 
> java.io.IOException: Failed to open transport to: host-ip:9160.
> 
> Any work around to use the stable loader from use 4.8.4(apache 2.1.12) to 
> apache 3.11.1
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

sstableloader from dse 4.8.4 to apache cassandra 3.11.1

[rajpal reddy]

Hello,

I’m trying to use sstablloader from dse 4.8.4( 2.1.12) to apache 3.11.1, i’m 
getting below error. but works fine when i use stableloader dse 5.1.2(apache 
3.11.0)
Could not retrieve endpoint ranges: 
java.io.IOException: Failed to open transport to: host-ip:9160.

Any work around to use the stable loader from use 4.8.4(apache 2.1.12) to 
apache 3.11.1


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re:

[Deniz Acay]

Yes there is a file like that. It contains a single line:

ADD:[/var/data/cassandra/data/[company_name]/[table_name]-3998ed90e01811e7820af142a4a9c0fd/mc-28797-big,0,8][3557304334]

On Tue, Jun 19, 2018 at 6:07 PM Jeff Jirsa  wrote:

> It's in the middle of trying to replay the unfinished compaction logs -
> there's a file in your data directories that will end with .log  - can you
> paste the contents somewhere like pastebin/gist?
>
>
>
> On Tue, Jun 19, 2018 at 7:42 AM, Deniz Acay  wrote:
>
>> I checked file system integrity and contents. Everything seems OK.
>>
>> Whatever the reason, that problem spread to all nodes at the same time,
>> so I believe it is not very likely for all volumes or any hardware to fail
>> at the same time.
>>
>> On Tue, Jun 19, 2018 at 2:25 PM Joshua Galbraith 
>> 
>> wrote:
>>
>>> Deniz,
>>>
>>> The assertion error you're seeing appears to be coming from this line:
>>>
>>> https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java#L63
>>>
>>> This file describes a LogReplicaSet as "A set of physical files on disk,
>>> [where] each file is an identical replica"
>>>
>>> https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L64
>>>
>>> This in reference the transaction logs:
>>>
>>> https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L45-L54
>>>
>>> Do you see anything odd about the log files, log directory, docker
>>> volume, or underlying EBS volume on the affected nodes?
>>>
>>> On Tue, Jun 19, 2018 at 7:13 AM, @Nandan@ <
>>> nandanpriyadarshi...@gmail.com> wrote:
>>>
 Check with Java Using version.

 On Tue, Jun 19, 2018 at 6:18 PM, Deniz Acay 
 wrote:

> Hello there,
>
> Let me get straight to the point. Yesterday our three node Cassandra
> production cluster had a problem and we could not find a solution yet.
> Before taking more radical actions, I would like to consult you about the
> issue.
>
> We are using Cassandra version 3.11.0. Cluster is living on AWS EC2
> nodes of type m4.2xlarge with 32 GBs of RAM. Each node Dockerized using
> host networking mode. Two EBS SSD volumes are attached to each node, 32GB
> for commit logs (io1) and 4TB for data directory (gp2). We have been
> running smoothly for 7 months and filled %55 of data directory on each 
> node.
> Now our C* nodes fail during bootstrapping phase. Let me paste the
> logs from system.log file from start to the time of error:
>
> INFO  [main] 2018-06-19 09:51:32,726 YamlConfigurationLoader.java:89 -
> Configuration location:
> file:/opt/apache-cassandra-3.11.0/conf/cassandra.yaml
> INFO  [main] 2018-06-19 09:51:32,954 Config.java:481 - Node
> configuration:[allocate_tokens_for_keyspace=botanalytics;
> authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer;
> auto_bootstrap=false; auto_snapshot=true; back_pressure_enabled=false;
> back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> broadcast_address=null; broadcast_rpc_address=null;
> buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000;
> cdc_enabled=false; cdc_free_space_check_interval_ms=250;
> cdc_raw_directory=/var/data/cassandra/cdc_raw; cdc_total_space_in_mb=0;
> client_encryption_options=; cluster_name=Botanalytics 
> Production;
> column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
> commit_failure_policy=stop_commit; commitlog_compression=null;
> commitlog_directory=/var/data/cassandra_commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=8192;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=1600; concurrent_compactors=null;
> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=64;
> counter_cache_keys_to_save=2147483647 <(214)%20748-3647>;
> counter_cache_save_period=7200; counter_cache_size_in_mb=null;
> counter_write_request_timeout_in_ms=5000;
> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@662b4c69;
> disk_access_mode=auto; disk_failure_policy=best_effort;
> disk_optimization_estimate_percentile=0.95;
> disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
> 

Re: RE: RE: [EXTERNAL] Cluster is unbalanced

[learner dba]

 Hi Sean,
Here is create table:

CREATE TABLE ks.cf (

    id text PRIMARY KEY,

    accessdata blob

) WITH bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''

    AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';
Nodetool status: 
Datacenter: dc1

===

Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address     Load       Tokens       Owns (effective)  Host ID               
                Rack

UN  x   20.66 GiB  256          61.4%             
f4f54949-83c9-419b-9a43-cb630b36d8c2  RAC1

UN  x  65.77 GiB  256          59.3%             
3db430ae-45ef-4746-a273-bc1f66ac8981  RAC1

UN  xx  60.58 GiB  256          58.4%             
1f23e869-1823-4b75-8d3e-f9b32acba9a6  RAC1

UN  x  47.08 GiB  256          57.5%             
7aca9a36-823f-4185-be44-c1464a799084  RAC1

UN  x  51.47 GiB  256          63.4%             
18cff010-9b83-4cf8-9dc2-f05ac63df402  RAC1

Datacenter: dc2



Status=Up/Down

|/ State=Normal/Leaving/Joining/Moving

--  Address     Load       Tokens       Owns (effective)  Host ID               
                Rack

UN     24.37 GiB  256          59.5%             
1b694180-210a-4b75-8f2a-748f4a5b6a3d  RAC1

UN  x 30.76 GiB  256          56.7%             
597bac04-c57a-4487-8924-72e171e45514  RAC1

UN    10.73 GiB  256          63.9%             
6e7e474e-e292-4433-afd4-372d30e0f3e1  RAC1

UN  xx 19.77 GiB  256          61.5%             
58751418-7b76-40f7-8b8f-a5bf8fe7d9a2  RAC1

UN  x  10.33 GiB  256          58.4%             
6d58d006-2095-449c-8c67-50e8cbdfe7a7  RAC1


cassandra-rackdc.properties:

dc=dc1
rack=RAC1 --> same in all nodes
cassandra.yaml:
num_tokens: 256
endpoint_snitch: GossipingPropertyFileSnitch
I can see cassandra-topology.properties, I believe it shouldn't be there with 
GossipPropertyFileSnitch. Can this file be causing any trouble in data 
distribution.

cat /opt/cassandra/conf/cassandra-topology.properties 

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.




# Cassandra Node IP=Data Center:Rack

192.168.1.100=DC1:RAC1

192.168.2.200=DC2:RAC2




10.0.0.10=DC1:RAC1

10.0.0.11=DC1:RAC1

10.0.0.12=DC1:RAC2




10.20.114.10=DC2:RAC1

10.20.114.11=DC2:RAC1




10.21.119.13=DC3:RAC1

10.21.119.10=DC3:RAC1




10.0.0.13=DC1:RAC2

10.21.119.14=DC3:RAC2

10.20.114.15=DC2:RAC2




# default for unknown nodes

default=DC1:r1




# Native IPv6 is supported, however you must escape the colon in the IPv6 
Address

# Also be sure to comment out JVM_OPTS="$JVM_OPTS 
-Djava.net.preferIPv4Stack=true"

# in cassandra-env.sh

fe80\:0\:0\:0\:202\:b3ff\:fe1e\:8329=DC1:RAC3






On Tuesday, June 19, 2018, 12:51:34 PM EDT, Durity, Sean R 
 wrote:  
 
  
You are correct that the cluster decides where data goes (based on the hash of 
the partition key). However, if you choose a “bad” partition key, you may not 
get good distribution of the data, because the hash is deterministic (it always 
goes to the same nodes/replicas). For example, if you have a partition key of a 
datetime, it is possible that there is more data written for a certain time 
period – thus a larger partition and an imbalance across the cluster. Choosing 
a “good” partition key is one of the most important decisions for a Cassandra 
table.
 
  
 
Also, I have seen the use of racks in the topology cause an imbalance in the 
“first” node of the rack.
 
  
 
To help you more, we would need the create table statement(s) for your keyspace 
and the topology of the cluster (like with nodetool status).
 
  
 
  
 
Sean Durity
 
From: learner dba 
Sent: Tuesday, June 19, 2018 9:50 AM
To: user@cassan

Re: How do you monitoring Cassandra Cluster?

[Romain Gérard]

Hi Felipe,

You can use this project https://github.com/criteo/cassandra_exporter if you 
are using Prometheus (Disclamer, I am one of the author of it).
There is included a Grafana dashboard that aggregate metrics per cluster for 
you, and in the "edit view" of each chart there are hidden queries that let you 
drill down by nodes.
In any case, you can look at the json in order to grasp how to aggregate 
queries.

Let me know if you need more information regarding the setup.
If you end-up using the project, give a star to the project it is always 
pleasant,
Regards,
Romain Gérard

On Jun 18 2018, at 3:25 pm, Felipe Esteves  
wrote:
>
> Hi, everyone,
>
> I'm running some tests to monitor Cassandra 3.x with jmx_exporter + 
> prometheus + grafana.
> I've managed to config it all and use the dashboard 
> https://grafana.com/dashboards/5408
>
> However, I still can't aggregate metrics from all my cluster, just nodes 
> individually.
> Any tips on how to do that?
>
> Also, Opscenter gives some datacenter aggregation, I think it comes from 
> nodetool as I didn't seen any metrics about that.
> Anyone having success on that?
>
> cheers!
> Em qua, 28 de jun de 2017 às 19:43, Petrus Gomes  (mailto:petru...@gmail.com)> escreveu:
> > I'm using JMX+Prometheus and Grafana.
> > JMX = https://github.com/prometheus/jmx_exporter
> > Prometheus + Grafana = https://prometheus.io/docs/visualization/grafana/
> >
> > There are some dashboard examples like that: 
> > https://grafana.com/dashboards/371
> > Looks good.
> >
> >
> > Thanks,
> > Petrus Silva
> >
> >
> >
> >
> > On Wed, Jun 28, 2017 at 5:55 AM, Peng Xiao <2535...@qq.com 
> > (mailto:2535...@qq.com)> wrote:
> > > Dear All,
> > >
> > > we are currently using Cassandra 2.1.13,and it has grown to 5TB size with 
> > > 32 nodes in one DC.
> > > For monitoring,opsCenter does not send alarm and not free in higher 
> > > version.so we have to use a simple JMX+Zabbix template.And we plan to use 
> > > Jolokia+JMX2Graphite to draw the metrics chart now.
> > >
> > > Could you please advise?
> > >
> > > Thanks,
> > > Henry
> >
> >
> >
> >
> >
> > Esta mensagem pode conter informações confidenciais e somente o indivíduo 
> > ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta 
> > da mensagem não acarreta a perda de sua confidencialidade. Caso esta 
> > mensagem tenha sido recebida por engano, solicitamos que o fato seja 
> > comunicado ao remetente e que a mensagem seja eliminada de seu sistema 
> > imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, 
> > revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de 
> > comunicação sujeito a monitoramento.
> > This message may include confidential information and only the intended 
> > addresses have the right to use it as is, or any part of it. A wrong 
> > transmission does not break its confidentiality. If you've received it 
> > because of a mistake or erroneous transmission, please notify the sender 
> > and delete it from your system immediately. This communication environment 
> > is controlled and monitored.
> > B2W Digital
> --
> Felipe Esteves
>
> Tecnologia
> felipe.este...@b2wdigital.com (mailto:seu.em...@b2wdigital.com)
> Tel.: (21) 3504-7162 ramal 57162

RE: RE: [EXTERNAL] Cluster is unbalanced

[Durity, Sean R]

You are correct that the cluster decides where data goes (based on the hash of 
the partition key). However, if you choose a “bad” partition key, you may not 
get good distribution of the data, because the hash is deterministic (it always 
goes to the same nodes/replicas). For example, if you have a partition key of a 
datetime, it is possible that there is more data written for a certain time 
period – thus a larger partition and an imbalance across the cluster. Choosing 
a “good” partition key is one of the most important decisions for a Cassandra 
table.

Also, I have seen the use of racks in the topology cause an imbalance in the 
“first” node of the rack.

To help you more, we would need the create table statement(s) for your keyspace 
and the topology of the cluster (like with nodetool status).


Sean Durity
From: learner dba 
Sent: Tuesday, June 19, 2018 9:50 AM
To: user@cassandra.apache.org
Subject: Re: RE: [EXTERNAL] Cluster is unbalanced

We do not chose the node where partition will go. I thought it is snitch's role 
to chose replica nodes. Even the partition size does not vary on our largest 
column family:

Percentile  SSTables Write Latency  Read LatencyPartition Size  
  Cell Count

  (micros)  (micros)   (bytes)

50% 0.00 17.08 61.21  3311  
   1

75% 0.00 20.50 88.15  3973  
   1

95% 0.00 35.43105.78  3973  
   1

98% 0.00 42.51126.93  3973  
   1

99% 0.00 51.01126.93  3973  
   1

Min 0.00  3.97 17.0961

Max 0.00 73.46126.93 11864  
   1

We are kinda stuck here to identify, what could be causing this un-balance.

On Tuesday, June 19, 2018, 7:15:28 AM EDT, Joshua Galbraith 
 wrote:


>If it was partition key issue, we would see similar number of partition keys 
>across nodes. If we look closely number of keys across nodes vary a lot.

I'm not sure about that, is it possible you're writing more new partitions to 
some nodes even though each node owns the same number of tokens?

[Image removed by sender.]

On Mon, Jun 18, 2018 at 6:07 PM, learner dba 
mailto:cassandra...@yahoo.com.invalid>> wrote:
Hi Sean,

Are you using any rack aware topology? --> we are using gossip file
Are you using any rack aware topology? --> we are using gossip file
 What are your partition keys? --> Partition key is uniq
Is it possible that your partition keys do not divide up as cleanly as you 
would like across the cluster because the data is not evenly distributed (by 
partition key)?  --> No, we verified it.

If it was partition key issue, we would see similar number of partition keys 
across nodes. If we look closely number of keys across nodes vary a lot.


Number of partitions (estimate): 3142552
Number of partitions (estimate): 15625442
Number of partitions (estimate): 15244021
Number of partitions (estimate): 9592992
Number of partitions (estimate): 15839280





On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R 
mailto:sean_r_dur...@homedepot.com>> wrote:



Are you using any rack aware topology? What are your partition keys? Is it 
possible that your partition keys do not divide up as cleanly as you would like 
across the cluster because the data is not evenly distributed (by partition 
key)?





Sean Durity

lord of the (C*) rings (Staff Systems Engineer – Cassandra)

MTC 2250

#cassandra - for the latest news and updates



From: learner dba mailto:cassandra...@yahoo.com>. 
INVALID>
Sent: Monday, June 18, 2018 2:06 PM
To: User 
cassandra.apache.org
 mailto:user@cassandra.apache.org>>
Subject: [EXTERNAL] Cluster is unbalanced



Hi,



Data volume varies a lot in our two DC cluster:

 Load   Tokens   Owns

 20.01 GiB  256  ?

 65.32 GiB  256  ?

 60.09 GiB  256  ?

 46.95 GiB  256  ?

 50.73 GiB  256  ?

kaiprodv2

=

/Leaving/Joining/Moving

 Load   Tokens   Owns

 25.19 GiB  256  ?

 30.26 GiB  256  ?

 9.82 GiB   256  ?

 20.54 GiB  256  ?

 9.7 GiB256  ?



I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on 
heavier nodes instead of decreasing. Based on nodetool cfstats, I can see 
partition keys on each node varies a lot:



Number of partitions (estimate): 3142552

Number of partitions (estimate): 15625442

Number of partitions (estimate): 15244021

RE: [EXTERNAL] Re: Tombstone

[Durity, Sean R]

This sounds like a queue pattern, which is typically an anti-pattern for 
Cassandra. I would say that it is very difficult to get the access patterns, 
tombstones, and everything else lined up properly to solve a queue problem.


Sean Durity

From: Abhishek Singh 
Sent: Tuesday, June 19, 2018 10:41 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: Tombstone

   The Partition key is made of datetime(basically date 
truncated to hour) and bucket.I think your RCA may be correct since we are 
deleting the partition rows one by one not in a batch files maybe overlapping 
for the particular partition.A scheduled thread picks the rows for a partition 
based on current datetime and bucket number and checks whether for each row the 
entiry is past due or not, if yes we trigger a event and remove the entry.



On Tue 19 Jun, 2018, 7:58 PM Jeff Jirsa, 
mailto:jji...@gmail.com>> wrote:
The most likely explanation is tombstones in files that won’t be collected as 
they potentially overlap data in other files with a lower timestamp (especially 
true if your partition key doesn’t change and you’re writing and deleting data 
within a partition)

--
Jeff Jirsa


> On Jun 19, 2018, at 3:28 AM, Abhishek Singh 
> mailto:abh23...@gmail.com>> wrote:
>
> Hi all,
>We using Cassandra for storing events which are time series based 
> for batch processing once a particular batch based on hour is processed we 
> delete the entries but we were left with almost 18% deletes marked as 
> Tombstones.
>  I ran compaction on the particular CF tombstone didn't come 
> down.
> Can anyone suggest what is the optimal tunning/recommended 
> practice used for compaction strategy and GC_grace period with 100k entries 
> and deletes every hour.
>
> Warm Regards
> Abhishek Singh

-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org

Re: 3.11.2 memory leak

[Abdul Patel]

Hi All,

Do we kmow whats the stable version for now if u wish to upgrade ?

On Tuesday, June 5, 2018, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Jeff,
>
>
>
> FWIW, when talking about https://issues.apache.org/
> jira/browse/CASSANDRA-13929, there is a patch available since March
> without getting further attention.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* Jeff Jirsa [mailto:jji...@gmail.com]
> *Sent:* Dienstag, 05. Juni 2018 00:51
> *To:* cassandra 
> *Subject:* Re: 3.11.2 memory leak
>
>
>
> There have been a few people who have reported it, but nobody (yet) has
> offered a patch to fix it. It would be good to have a reliable way to
> repro, and/or an analysis of a heap dump demonstrating the problem (what's
> actually retained at the time you're OOM'ing).
>
>
>
> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel  wrote:
>
> Hi All,
>
>
>
> I recently upgraded my non prod cluster from 3.10 to 3.11.2.
>
> It was working fine for a 1.5 weeks then suddenly nodetool info startee
> reporting 80% and more memory consumption.
>
> Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4
> nodes of cluster-single DC.
>
> Now after 8 days i again see 80% + usage and its 16gb and above ..which we
> never saw before .
>
> Seems like memory leak bug?
>
> Does anyone has any idea ? Our 3.11.2 release rollout has been halted
> because of this.
>
> If not 3.11.2 whats the next best stable release we have now?
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

Re:

[Jeff Jirsa]

It's in the middle of trying to replay the unfinished compaction logs -
there's a file in your data directories that will end with .log  - can you
paste the contents somewhere like pastebin/gist?



On Tue, Jun 19, 2018 at 7:42 AM, Deniz Acay  wrote:

> I checked file system integrity and contents. Everything seems OK.
>
> Whatever the reason, that problem spread to all nodes at the same time, so
> I believe it is not very likely for all volumes or any hardware to fail at
> the same time.
>
> On Tue, Jun 19, 2018 at 2:25 PM Joshua Galbraith 
> 
> wrote:
>
>> Deniz,
>>
>> The assertion error you're seeing appears to be coming from this line:
>> https://github.com/apache/cassandra/blob/cassandra-3.11.
>> 0/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java#L63
>>
>> This file describes a LogReplicaSet as "A set of physical files on disk,
>> [where] each file is an identical replica"
>> https://github.com/apache/cassandra/blob/cassandra-3.11.
>> 0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L64
>>
>> This in reference the transaction logs:
>> https://github.com/apache/cassandra/blob/cassandra-3.11.
>> 0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L45-L54
>>
>> Do you see anything odd about the log files, log directory, docker
>> volume, or underlying EBS volume on the affected nodes?
>>
>> On Tue, Jun 19, 2018 at 7:13 AM, @Nandan@ > > wrote:
>>
>>> Check with Java Using version.
>>>
>>> On Tue, Jun 19, 2018 at 6:18 PM, Deniz Acay  wrote:
>>>
 Hello there,

 Let me get straight to the point. Yesterday our three node Cassandra
 production cluster had a problem and we could not find a solution yet.
 Before taking more radical actions, I would like to consult you about the
 issue.

 We are using Cassandra version 3.11.0. Cluster is living on AWS EC2
 nodes of type m4.2xlarge with 32 GBs of RAM. Each node Dockerized using
 host networking mode. Two EBS SSD volumes are attached to each node, 32GB
 for commit logs (io1) and 4TB for data directory (gp2). We have been
 running smoothly for 7 months and filled %55 of data directory on each 
 node.
 Now our C* nodes fail during bootstrapping phase. Let me paste the logs
 from system.log file from start to the time of error:

 INFO  [main] 2018-06-19 09:51:32,726 YamlConfigurationLoader.java:89 -
 Configuration location: file:/opt/apache-cassandra-3.
 11.0/conf/cassandra.yaml
 INFO  [main] 2018-06-19 09:51:32,954 Config.java:481 - Node
 configuration:[allocate_tokens_for_keyspace=botanalytics;
 authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer;
 auto_bootstrap=false; auto_snapshot=true; back_pressure_enabled=false;
 back_pressure_strategy=org.apache.cassandra.net.
 RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST};
 batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5;
 batchlog_replay_throttle_in_kb=1024; broadcast_address=null;
 broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true;
 cas_contention_timeout_in_ms=1000; cdc_enabled=false;
 cdc_free_space_check_interval_ms=250; 
 cdc_raw_directory=/var/data/cassandra/cdc_raw;
 cdc_total_space_in_mb=0; client_encryption_options=;
 cluster_name=Botanalytics Production; column_index_cache_size_in_kb=2;
 column_index_size_in_kb=64; commit_failure_policy=stop_commit;
 commitlog_compression=null; 
 commitlog_directory=/var/data/cassandra_commitlog;
 commitlog_max_compression_buffers_in_pool=3;
 commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
 commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
 commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=8192;
 compaction_large_partition_warning_threshold_mb=100;
 compaction_throughput_mb_per_sec=1600; concurrent_compactors=null;
 concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
 concurrent_reads=32; concurrent_replicates=null; concurrent_writes=64;
 counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
 counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
 credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
 credentials_validity_in_ms=2000; cross_node_timeout=false;
 data_file_directories=[Ljava.lang.String;@662b4c69;
 disk_access_mode=auto; disk_failure_policy=best_effort;
 disk_optimization_estimate_percentile=0.95;
 disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
 dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
 dynamic_snitch_reset_interval_in_ms=60; 
 dynamic_snitch_update_interval_in_ms=100;
 enable_scripted_user_defined_functions=false;
 enable_user_defined_functions=false; 
 enable_user_defined_functions_threads=true;
 encryption_options=null; endpoint_snitch=Ec2Snitch;
 file_cache_size_in_mb=n

Re:

[Deniz Acay]

I checked file system integrity and contents. Everything seems OK.

Whatever the reason, that problem spread to all nodes at the same time, so
I believe it is not very likely for all volumes or any hardware to fail at
the same time.

On Tue, Jun 19, 2018 at 2:25 PM Joshua Galbraith
 wrote:

> Deniz,
>
> The assertion error you're seeing appears to be coming from this line:
>
> https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java#L63
>
> This file describes a LogReplicaSet as "A set of physical files on disk,
> [where] each file is an identical replica"
>
> https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L64
>
> This in reference the transaction logs:
>
> https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L45-L54
>
> Do you see anything odd about the log files, log directory, docker volume,
> or underlying EBS volume on the affected nodes?
>
> On Tue, Jun 19, 2018 at 7:13 AM, @Nandan@ 
> wrote:
>
>> Check with Java Using version.
>>
>> On Tue, Jun 19, 2018 at 6:18 PM, Deniz Acay  wrote:
>>
>>> Hello there,
>>>
>>> Let me get straight to the point. Yesterday our three node Cassandra
>>> production cluster had a problem and we could not find a solution yet.
>>> Before taking more radical actions, I would like to consult you about the
>>> issue.
>>>
>>> We are using Cassandra version 3.11.0. Cluster is living on AWS EC2
>>> nodes of type m4.2xlarge with 32 GBs of RAM. Each node Dockerized using
>>> host networking mode. Two EBS SSD volumes are attached to each node, 32GB
>>> for commit logs (io1) and 4TB for data directory (gp2). We have been
>>> running smoothly for 7 months and filled %55 of data directory on each node.
>>> Now our C* nodes fail during bootstrapping phase. Let me paste the logs
>>> from system.log file from start to the time of error:
>>>
>>> INFO  [main] 2018-06-19 09:51:32,726 YamlConfigurationLoader.java:89 -
>>> Configuration location:
>>> file:/opt/apache-cassandra-3.11.0/conf/cassandra.yaml
>>> INFO  [main] 2018-06-19 09:51:32,954 Config.java:481 - Node
>>> configuration:[allocate_tokens_for_keyspace=botanalytics;
>>> authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer;
>>> auto_bootstrap=false; auto_snapshot=true; back_pressure_enabled=false;
>>> back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
>>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>> broadcast_address=null; broadcast_rpc_address=null;
>>> buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000;
>>> cdc_enabled=false; cdc_free_space_check_interval_ms=250;
>>> cdc_raw_directory=/var/data/cassandra/cdc_raw; cdc_total_space_in_mb=0;
>>> client_encryption_options=; cluster_name=Botanalytics Production;
>>> column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
>>> commit_failure_policy=stop_commit; commitlog_compression=null;
>>> commitlog_directory=/var/data/cassandra_commitlog;
>>> commitlog_max_compression_buffers_in_pool=3;
>>> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
>>> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
>>> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=8192;
>>> compaction_large_partition_warning_threshold_mb=100;
>>> compaction_throughput_mb_per_sec=1600; concurrent_compactors=null;
>>> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
>>> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=64;
>>> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
>>> credentials_validity_in_ms=2000; cross_node_timeout=false;
>>> data_file_directories=[Ljava.lang.String;@662b4c69;
>>> disk_access_mode=auto; disk_failure_policy=best_effort;
>>> disk_optimization_estimate_percentile=0.95;
>>> disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
>>> dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
>>> dynamic_snitch_reset_interval_in_ms=60;
>>> dynamic_snitch_update_interval_in_ms=100;
>>> enable_scripted_user_defined_functions=false;
>>> enable_user_defined_functions=false;
>>> enable_user_defined_functions_threads=true; encryption_options=null;
>>> endpoint_snitch=Ec2Snitch; file_cache_size_in_mb=null;
>>> gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000;
>>> hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true;
>>> hinted_handoff_throttle_in_kb=1024; hints_compression=null;
>>> hints_directory=null; hints_flush_period_in_ms=1;
>>> incremental_backups=false; index_interval=null;
>>> index_summary_capacity_in_mb=null;
>>> index_summary_resize_int

Re: Tombstone

[Abhishek Singh]

   The Partition key is made of datetime(basically date
truncated to hour) and bucket.I think your RCA may be correct since we are
deleting the partition rows one by one not in a batch files maybe
overlapping for the particular partition.A scheduled thread picks the rows
for a partition based on current datetime and bucket number and checks
whether for each row the entiry is past due or not, if yes we trigger a
event and remove the entry.



On Tue 19 Jun, 2018, 7:58 PM Jeff Jirsa,  wrote:

> The most likely explanation is tombstones in files that won’t be collected
> as they potentially overlap data in other files with a lower timestamp
> (especially true if your partition key doesn’t change and you’re writing
> and deleting data within a partition)
>
> --
> Jeff Jirsa
>
>
> > On Jun 19, 2018, at 3:28 AM, Abhishek Singh  wrote:
> >
> > Hi all,
> >We using Cassandra for storing events which are time series
> based for batch processing once a particular batch based on hour is
> processed we delete the entries but we were left with almost 18% deletes
> marked as Tombstones.
> >  I ran compaction on the particular CF tombstone didn't
> come down.
> > Can anyone suggest what is the optimal tunning/recommended
> practice used for compaction strategy and GC_grace period with 100k entries
> and deletes every hour.
> >
> > Warm Regards
> > Abhishek Singh
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re:

[Jeff Jirsa]

Just assume that the rows you read in a page all end up in the heap at the same 
time

If you’re reading 1000 rows of 100 bytes, no big deal, you’ve got 100kb per 
read thread on the heap

If you’re reading 100 1mb rows, now you’ve got 100MB per thread on the heap

Assuming an 8gb heap with 2gb young gen size, the first example is probably no 
problem even with dozens of concurrent reads, but the second will trigger a 
young gc every 10-15 reads (could be promotion, depending on how many 
concurrent reads you’re doing). 




-- 
Jeff Jirsa


> On Jun 19, 2018, at 1:53 AM, Vsevolod Filaretov  wrote:
> 
> Kurt, thank you very much for your answer! Your remark on GC totally changed 
> my thoughts on cassandra resources usage.
> 
> So.. more questions to the respective audience underway.
> 
> What is generally considered as 
> 
> 1) "too large" page size, 
> 2)"large" page size
> 3) "normal conditions" page size?
> 
> How exactly fetch size affects CPU? Can too large page size provoke severe 
> CPU usage for constant GC, thus affecting Cassandra performance on read 
> requests (because CPU basically doesn't work on other tasks, while it's 
> constantly GCing)?
> 
> Thank you all very much!
> 
> пн, 18 июн. 2018 г., 14:28 kurt greaves :
>>> 1) Am I correct to assume that the larger page size some user session has 
>>> set - the larger portion of cluster/coordinator node resources will be 
>>> hogged by the corresponding session?
>>> 2) Do I understand correctly that page size (imagine we have no timeout 
>>> settings) is limited by RAM and iops which I want to hand down to a single 
>>> user session?
>> Yes for both of the above. More rows will be pulled into memory 
>> simultaneously with a larger page size, thus using more memory and IO. 
>> 
>>> 3) Am I correct to assume that the page size/read request timeout allowance 
>>> I set is direct representation of chance to lock some node to single user's 
>>> requests?
>> Concurrent reads can occur on a node, so it shouldn't "lock" the node to a 
>> single users request. However you can overload the node, which may be 
>> effectively the same thing. Don't set page sizes too high, otherwise the 
>> coordinator of the query will end up doing a lot of GC. 
>> 
>> 

Re: Tombstone

[Jeff Jirsa]

The most likely explanation is tombstones in files that won’t be collected as 
they potentially overlap data in other files with a lower timestamp (especially 
true if your partition key doesn’t change and you’re writing and deleting data 
within a partition)

-- 
Jeff Jirsa


> On Jun 19, 2018, at 3:28 AM, Abhishek Singh  wrote:
> 
> Hi all,
>We using Cassandra for storing events which are time series based 
> for batch processing once a particular batch based on hour is processed we 
> delete the entries but we were left with almost 18% deletes marked as 
> Tombstones.
>  I ran compaction on the particular CF tombstone didn't come 
> down.
> Can anyone suggest what is the optimal tunning/recommended 
> practice used for compaction strategy and GC_grace period with 100k entries 
> and deletes every hour.
> 
> Warm Regards
> Abhishek Singh

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Timestamp on hints file and system.hints table data

[learner dba]

 Kurt,
There are no errors in system.log. Just messages about hints sent and played. 
We will try stop and delete solution and let you know if that fixed the 
problem. But my curiosity is more about timestamp on hints files, why do we 
have files with back date; even though I deleted files manually.
On Monday, June 18, 2018, 9:12:16 PM EDT, kurt greaves 
 wrote:  
 
 Send through some examples (and any errors)? Sounds like the file might be 
corrupt. Not that there's much you can do about that. You can try stopping C*, 
deleting the file, then starting C* again. You'll have to repair, assuming you 
haven't repaired already since that hint file was created.
On 18 June 2018 at 13:56, learner dba  wrote:

 Yes Kurt, system log is flooded with hints sent and replayed messages. 
On Monday, June 18, 2018, 7:30:34 AM EDT, kurt greaves 
 wrote:  
 
 Not sure what to make of that. Are there any log messages regarding the file 
and replaying hints? Sounds like maybe it's corrupt (although not sure why it 
keeps getting rewritten).

On 14 June 2018 at 13:19, Nitan Kainth  wrote:

Kurt,
Hint file matches UUID matches with another node in the cluster:

-rw-r--r--. 1 root root  6848246 May 13 23:37 1b694180-210a-4b75-8f2a- 
748f4a5b6a3d-1526254645089-1. hints


/opt/cassandra/bin/nodetool status |grep 1b694180

UN  x.x.x.   23.77 GiB  256          ?       1b694180-210a-4b75-8f2a- 
748f4a5b6a3d  RAC1



On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves  wrote:

Does the UUID on the filename correspond with a UUID in nodetool status?
Sounds to me like it could be something weird with an old node that no longer 
exists, although hints for old nodes are meant to be cleaned up.
On 14 June 2018 at 01:54, Nitan Kainth  wrote:

Kurt,
No node is down for months. And yes, I am surprised to look at Unix timestamp 
on files.


On Jun 13, 2018, at 6:41 PM, kurt greaves  wrote:


system.hints is not used in Cassandra 3. Can't explain the files though, are 
you referring to the files timestamp or the Unix timestamp in the file name? Is 
there a node that's been down for several months?

On Wed., 13 Jun. 2018, 23:41 Nitan Kainth,  wrote:

Hi,
I observed a strange behavior about stored hints. 
Time stamp of hints file shows several months old. I deleted them and saw new 
hints files created with same old date. Why is that?
Also, I see hints files on disk but if I query system.hints table, it shows 0 
rows. Why system.hints is not populated?
Version 3.11-1







  

  

Re: RE: [EXTERNAL] Cluster is unbalanced

[learner dba]

 We do not chose the node where partition will go. I thought it is snitch's 
role to chose replica nodes. Even the partition size does not vary on our 
largest column family:


Percentile  SSTables     Write Latency      Read Latency    Partition Size      
  Cell Count

                              (micros)          (micros)           (bytes)      
            

50%             0.00             17.08             61.21              3311      
           1

75%             0.00             20.50             88.15              3973      
           1

95%             0.00             35.43            105.78              3973      
           1

98%             0.00             42.51            126.93              3973      
           1

99%             0.00             51.01            126.93              3973      
           1

Min             0.00              3.97             17.09                61      
           0

Max             0.00             73.46            126.93             11864      
           1

We are kinda stuck here to identify, what could be causing this un-balance.
On Tuesday, June 19, 2018, 7:15:28 AM EDT, Joshua Galbraith 
 wrote:  
 
 >If it was partition key issue, we would see similar number of partition keys 
 >across nodes. If we look closely number of keys across nodes vary a lot.

I'm not sure about that, is it possible you're writing more new partitions to 
some nodes even though each node owns the same number of tokens?

On Mon, Jun 18, 2018 at 6:07 PM, learner dba  
wrote:

 Hi Sean,
Are you using any rack aware topology? --> we are using gossip file
 Are you using any rack aware topology? --> we are using gossip file What are 
your partition keys? --> Partition key is uniqIs it possible that your 
partition keys do not divide up as cleanly as you would like across the cluster 
because the data is not evenly distributed (by partition key)?  --> No, we 
verified it.
If it was partition key issue, we would see similar number of partition keys 
across nodes. If we look closely number of keys across nodes vary a lot.

Number of partitions (estimate): 3142552Number of partitions (estimate): 
15625442Number of partitions (estimate): 15244021Number of partitions 
(estimate): 9592992Number of partitions (estimate): 15839280




On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R 
 wrote:  
 
 
Are you using any rack aware topology? What are your partition keys? Is it 
possible that your partition keys do not divide up as cleanly as you would like 
across the cluster because the data is not evenly distributed (by partition 
key)?
 
  
 
  
 
Sean Durity
 
lord of the (C*) rings (Staff Systems Engineer – Cassandra)
 
MTC 2250
 
#cassandra - for the latest news and updates
 
  
 
From: learner dba 
Sent: Monday, June 18, 2018 2:06 PM
To: User cassandra.apache.org 
Subject: [EXTERNAL] Cluster is unbalanced
 
  
 
Hi,
 
  
 
Data volume varies a lot in our two DC cluster:
 
 Load     Tokens     Owns 
 
 20.01 GiB 256         ?     
 
 65.32 GiB 256         ?     
 
 60.09 GiB 256         ?     
 
 46.95 GiB 256         ?     
 
 50.73 GiB 256         ?     
 
kaiprodv2
 
=
 
/Leaving/Joining/Moving
 
 Load     Tokens     Owns 
 
 25.19 GiB 256         ?     
 
 30.26 GiB 256         ?     
 
 9.82 GiB 256         ?     
 
 20.54 GiB 256         ?     
 
 9.7 GiB    256         ?     
 
  
 
I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on 
heavier nodes instead of decreasing. Based on nodetool cfstats, I can see 
partition keys on each node varies a lot:
 
  
 
Number of partitions (estimate): 3142552
 
Number of partitions (estimate): 15625442
 
Number of partitions (estimate): 15244021
 
Number of partitions (estimate): 9592992
 
Number of partitions (estimate): 15839280
 
  
 
How can I diagnose this imbalance further?
 
  
   



-- 
Joshua Galbraith | Senior Software Engineer | New Relic
C: 907-209-1208 | jgalbra...@newrelic.com
  

Re: Cassandra Client Program not Working with NettySSLOptions

[Jonathan Haddad]

Is the server configured to use encryption?

On Tue, Jun 19, 2018 at 3:59 AM Jahar Tyagi  wrote:

> Hi,
>
> I referred to this link
> https://docs.datastax.com/en/developer/java-driver/3.0/manual/ssl/
>   to
> implement a simple Cassandra client using datastax driver 3.0.0 on SSL with
> OpenSSL options but unable to run it.
>
> Getting generic exception as " 
> *com.datastax.driver.core.exceptions.NoHostAvailableException"
> *at line
> mySession = myCluster.connect();
>
> *Code snippet to setup cluster connection is below.*
>
> public void connectToCluster()
> {
> String[] theCassandraHosts = {"myip"};
> myCluster =
>
> Cluster.builder().withSSL(getSSLOption()).withReconnectionPolicy(new
> ConstantReconnectionPolicy(2000)).addContactPoints(theCassandraHosts).withPort(10742)
> .withCredentials("username",
> "password").withLoadBalancingPolicy(DCAwareRoundRobinPolicy.builder().build())
> .withSocketOptions(new
> SocketOptions().setConnectTimeoutMillis(800).setKeepAlive(true)).build();
> try {
> mySession = myCluster.connect();
> }
> catch(Exception e) {
> e.printStackTrace();
> }
> System.out.println("Session Established");
> }
>
>
>  private SSLOptions getSSLOption()
> {
> InputStream trustStore = null;
> try
> {
> String theTrustStorePath =
> "/var/opt/SecureInterface/myTrustStore.jks";
> String theTrustStorePassword = "mypassword";
> List theCipherSuites = new ArrayList();
> theCipherSuites.add("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384");
> KeyStore ks = KeyStore.getInstance("JKS");
> *trustStore = new FileInputStream(theTrustStorePath);*
> ks.load(trustStore, theTrustStorePassword.toCharArray());
> TrustManagerFactory tmf =
> TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
> tmf.init(ks);
> SslContextBuilder builder =
> SslContextBuilder.forClient()
> .sslProvider(SslProvider.OPENSSL)
> .trustManager(tmf)
> .ciphers(theCipherSuites)
> // only if you use client authentication
> .keyManager(new
> File("/var/opt/SecureInterface/keystore/Cass.crt"),
> new
> File("/var/opt/vs/SecureInterface/keystore/Cass_enc.key"));
> SSLOptions sslOptions = new NettySSLOptions(builder.build());
> return sslOptions;
> }
> catch (Exception e)
> {
> e.printStackTrace();
> }
> finally
> {
> try
> {
> trustStore.close();
> }
> catch (IOException e)
> {
> e.printStackTrace();
> }
> }
> return null;
> }
>
> Cassandra server is running fine with client and server encryption
> options. Moreover  I am able to run my client using JdkSSLOptions but have
> problem with NettySSLOptions.
>
> Has anyone implemented the  NettySSLOptions for Cassandra client
> application?
>
>
> Regards,
> Jahar Tyagi
>
-- 
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade

Re:

[Joshua Galbraith]

Deniz,

The assertion error you're seeing appears to be coming from this line:
https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogReplicaSet.java#L63

This file describes a LogReplicaSet as "A set of physical files on disk,
[where] each file is an identical replica"
https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L64

This in reference the transaction logs:
https://github.com/apache/cassandra/blob/cassandra-3.11.0/src/java/org/apache/cassandra/db/lifecycle/LogFile.java#L45-L54

Do you see anything odd about the log files, log directory, docker volume,
or underlying EBS volume on the affected nodes?

On Tue, Jun 19, 2018 at 7:13 AM, @Nandan@ 
wrote:

> Check with Java Using version.
>
> On Tue, Jun 19, 2018 at 6:18 PM, Deniz Acay  wrote:
>
>> Hello there,
>>
>> Let me get straight to the point. Yesterday our three node Cassandra
>> production cluster had a problem and we could not find a solution yet.
>> Before taking more radical actions, I would like to consult you about the
>> issue.
>>
>> We are using Cassandra version 3.11.0. Cluster is living on AWS EC2
>> nodes of type m4.2xlarge with 32 GBs of RAM. Each node Dockerized using
>> host networking mode. Two EBS SSD volumes are attached to each node, 32GB
>> for commit logs (io1) and 4TB for data directory (gp2). We have been
>> running smoothly for 7 months and filled %55 of data directory on each node.
>> Now our C* nodes fail during bootstrapping phase. Let me paste the logs
>> from system.log file from start to the time of error:
>>
>> INFO  [main] 2018-06-19 09:51:32,726 YamlConfigurationLoader.java:89 -
>> Configuration location: file:/opt/apache-cassandra-3.1
>> 1.0/conf/cassandra.yaml
>> INFO  [main] 2018-06-19 09:51:32,954 Config.java:481 - Node
>> configuration:[allocate_tokens_for_keyspace=botanalytics;
>> authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer;
>> auto_bootstrap=false; auto_snapshot=true; back_pressure_enabled=false;
>> back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
>> factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>> broadcast_address=null; broadcast_rpc_address=null;
>> buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000;
>> cdc_enabled=false; cdc_free_space_check_interval_ms=250;
>> cdc_raw_directory=/var/data/cassandra/cdc_raw; cdc_total_space_in_mb=0;
>> client_encryption_options=; cluster_name=Botanalytics
>> Production; column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
>> commit_failure_policy=stop_commit; commitlog_compression=null;
>> commitlog_directory=/var/data/cassandra_commitlog;
>> commitlog_max_compression_buffers_in_pool=3;
>> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
>> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
>> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=8192;
>> compaction_large_partition_warning_threshold_mb=100;
>> compaction_throughput_mb_per_sec=1600; concurrent_compactors=null;
>> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
>> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=64;
>> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
>> credentials_validity_in_ms=2000; cross_node_timeout=false;
>> data_file_directories=[Ljava.lang.String;@662b4c69;
>> disk_access_mode=auto; disk_failure_policy=best_effort;
>> disk_optimization_estimate_percentile=0.95;
>> disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
>> dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
>> dynamic_snitch_reset_interval_in_ms=60;
>> dynamic_snitch_update_interval_in_ms=100; 
>> enable_scripted_user_defined_functions=false;
>> enable_user_defined_functions=false; 
>> enable_user_defined_functions_threads=true;
>> encryption_options=null; endpoint_snitch=Ec2Snitch;
>> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
>> gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[];
>> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
>> hints_compression=null; hints_directory=null; hints_flush_period_in_ms=1;
>> incremental_backups=false; index_interval=null;
>> index_summary_capacity_in_mb=null; 
>> index_summary_resize_interval_in_minutes=60;
>> initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200;
>> inter_dc_tcp_nodelay=false; internode_authenticator=null;
>> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
>> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
>> key_cache_save_period=14400; key_cache_size_in_mb=null;
>> listen_address=172.31.6.2

Re: RE: [EXTERNAL] Cluster is unbalanced

[Joshua Galbraith]

>If it was partition key issue, we would see similar number of partition
keys across nodes. If we look closely number of keys across nodes vary a
lot.

I'm not sure about that, is it possible you're writing more new partitions
to some nodes even though each node owns the same number of tokens?

On Mon, Jun 18, 2018 at 6:07 PM, learner dba  wrote:

> Hi Sean,
>
> Are you using any rack aware topology? --> we are using gossip file
> Are you using any rack aware topology? --> we are using gossip file
>  What are your partition keys? --> Partition key is uniq
> Is it possible that your partition keys do not divide up as cleanly as you
> would like across the cluster because the data is not evenly distributed
> (by partition key)?  --> No, we verified it.
>
> If it was partition key issue, we would see similar number of partition
> keys across nodes. If we look closely number of keys across nodes vary a
> lot.
>
>
> Number of partitions (estimate): 3142552
> Number of partitions (estimate): 15625442
> Number of partitions (estimate): 15244021
> Number of partitions (estimate): 9592992
> Number of partitions (estimate): 15839280
>
>
>
>
>
> On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
>
> Are you using any rack aware topology? What are your partition keys? Is it
> possible that your partition keys do not divide up as cleanly as you would
> like across the cluster because the data is not evenly distributed (by
> partition key)?
>
>
>
>
>
> Sean Durity
>
> lord of the (C*) rings (Staff Systems Engineer – Cassandra)
>
> MTC 2250
>
> #cassandra - for the latest news and updates
>
>
>
> *From:* learner dba 
> *Sent:* Monday, June 18, 2018 2:06 PM
> *To:* User cassandra.apache.org 
> *Subject:* [EXTERNAL] Cluster is unbalanced
>
>
>
> Hi,
>
>
>
> Data volume varies a lot in our two DC cluster:
>
>  Load   Tokens   Owns
>
>  20.01 GiB  256  ?
>
>  65.32 GiB  256  ?
>
>  60.09 GiB  256  ?
>
>  46.95 GiB  256  ?
>
>  50.73 GiB  256  ?
>
> kaiprodv2
>
> =
>
> /Leaving/Joining/Moving
>
>  Load   Tokens   Owns
>
>  25.19 GiB  256  ?
>
>  30.26 GiB  256  ?
>
>  9.82 GiB   256  ?
>
>  20.54 GiB  256  ?
>
>  9.7 GiB256  ?
>
>
>
> I ran clearsnapshot, garbagecollect and cleanup, but it increased the size
> on heavier nodes instead of decreasing. Based on nodetool cfstats, I can
> see partition keys on each node varies a lot:
>
>
>
> Number of partitions (estimate): 3142552
>
> Number of partitions (estimate): 15625442
>
> Number of partitions (estimate): 15244021
>
> Number of partitions (estimate): 9592992
>
> Number of partitions (estimate): 15839280
>
>
>
> How can I diagnose this imbalance further?
>
>
>



-- 
*Joshua Galbraith *| Senior Software Engineer | New Relic
C: 907-209-1208 | jgalbra...@newrelic.com

Re:

[@Nandan@]

Check with Java Using version.

On Tue, Jun 19, 2018 at 6:18 PM, Deniz Acay  wrote:

> Hello there,
>
> Let me get straight to the point. Yesterday our three node Cassandra
> production cluster had a problem and we could not find a solution yet.
> Before taking more radical actions, I would like to consult you about the
> issue.
>
> We are using Cassandra version 3.11.0. Cluster is living on AWS EC2 nodes
> of type m4.2xlarge with 32 GBs of RAM. Each node Dockerized using host
> networking mode. Two EBS SSD volumes are attached to each node, 32GB for
> commit logs (io1) and 4TB for data directory (gp2). We have been running
> smoothly for 7 months and filled %55 of data directory on each node.
> Now our C* nodes fail during bootstrapping phase. Let me paste the logs
> from system.log file from start to the time of error:
>
> INFO  [main] 2018-06-19 09:51:32,726 YamlConfigurationLoader.java:89 -
> Configuration location: file:/opt/apache-cassandra-3.
> 11.0/conf/cassandra.yaml
> INFO  [main] 2018-06-19 09:51:32,954 Config.java:481 - Node
> configuration:[allocate_tokens_for_keyspace=botanalytics; 
> authenticator=AllowAllAuthenticator;
> authorizer=AllowAllAuthorizer; auto_bootstrap=false; auto_snapshot=true;
> back_pressure_enabled=false; back_pressure_strategy=org.
> apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5,
> flow=FAST}; batch_size_fail_threshold_in_kb=50;
> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
> broadcast_address=null; broadcast_rpc_address=null; 
> buffer_pool_use_heap_if_exhausted=true;
> cas_contention_timeout_in_ms=1000; cdc_enabled=false;
> cdc_free_space_check_interval_ms=250; 
> cdc_raw_directory=/var/data/cassandra/cdc_raw;
> cdc_total_space_in_mb=0; client_encryption_options=;
> cluster_name=Botanalytics Production; column_index_cache_size_in_kb=2;
> column_index_size_in_kb=64; commit_failure_policy=stop_commit;
> commitlog_compression=null; commitlog_directory=/var/data/cassandra_commitlog;
> commitlog_max_compression_buffers_in_pool=3;
> commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
> commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
> commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=8192;
> compaction_large_partition_warning_threshold_mb=100;
> compaction_throughput_mb_per_sec=1600; concurrent_compactors=null;
> concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
> concurrent_reads=32; concurrent_replicates=null; concurrent_writes=64;
> counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
> credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
> credentials_validity_in_ms=2000; cross_node_timeout=false;
> data_file_directories=[Ljava.lang.String;@662b4c69;
> disk_access_mode=auto; disk_failure_policy=best_effort;
> disk_optimization_estimate_percentile=0.95; 
> disk_optimization_page_cross_chance=0.1;
> disk_optimization_strategy=ssd; dynamic_snitch=true;
> dynamic_snitch_badness_threshold=0.1; 
> dynamic_snitch_reset_interval_in_ms=60;
> dynamic_snitch_update_interval_in_ms=100; 
> enable_scripted_user_defined_functions=false;
> enable_user_defined_functions=false; 
> enable_user_defined_functions_threads=true;
> encryption_options=null; endpoint_snitch=Ec2Snitch;
> file_cache_size_in_mb=null; gc_log_threshold_in_ms=200;
> gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[];
> hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024;
> hints_compression=null; hints_directory=null; hints_flush_period_in_ms=1;
> incremental_backups=false; index_interval=null;
> index_summary_capacity_in_mb=null; 
> index_summary_resize_interval_in_minutes=60;
> initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200;
> inter_dc_tcp_nodelay=false; internode_authenticator=null;
> internode_compression=dc; internode_recv_buff_size_in_bytes=0;
> internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
> key_cache_save_period=14400; key_cache_size_in_mb=null;
> listen_address=172.31.6.233; listen_interface=null;
> listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
> max_hint_window_in_ms=1080; max_hints_delivery_threads=2;
> max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
> max_streaming_retries=3; max_value_size_in_mb=256;
> memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null;
> memtable_flush_writers=0; memtable_heap_space_in_mb=null;
> memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50;
> native_transport_max_concurrent_connections=-1; native_transport_max_
> concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256;
> native_transport_max_threads=128; native_transport_port=9042;
> native_transport_port_ssl=null; num_tokens=8; 
> otc_backlog_expiration_interval_ms=200;
> otc_coalescing_enough_coalesced_messages=8; ot

Re: Tombstone

[Evelyn Smith]

TimeWindowCompactionStrategy and don’t delete the data you should be relying on 
Cassandra to drop the SSTables once the data inside has expired.

THat 18% is probably waiting on gc_grace, this shouldn’t be an issue if you are 
letting TWCS drop the data rather then running deletes.

Regards,
Evelyn.

> On 19 Jun 2018, at 8:28 pm, Abhishek Singh  wrote:
> 
> Hi all,
>We using Cassandra for storing events which are time series based 
> for batch processing once a particular batch based on hour is processed we 
> delete the entries but we were left with almost 18% deletes marked as 
> Tombstones.
>  I ran compaction on the particular CF tombstone didn't come 
> down.
> Can anyone suggest what is the optimal tunning/recommended 
> practice used for compaction strategy and GC_grace period with 100k entries 
> and deletes every hour.
> 
> Warm Regards
> Abhishek Singh


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Cassandra Client Program not Working with NettySSLOptions

[Jahar Tyagi]

Hi,

I referred to this link
https://docs.datastax.com/en/developer/java-driver/3.0/manual/ssl/
  to
implement a simple Cassandra client using datastax driver 3.0.0 on SSL with
OpenSSL options but unable to run it.

Getting generic exception as "
*com.datastax.driver.core.exceptions.NoHostAvailableException"
*at line
mySession = myCluster.connect();

*Code snippet to setup cluster connection is below.*

public void connectToCluster()
{
String[] theCassandraHosts = {"myip"};
myCluster =

Cluster.builder().withSSL(getSSLOption()).withReconnectionPolicy(new
ConstantReconnectionPolicy(2000)).addContactPoints(theCassandraHosts).withPort(10742)
.withCredentials("username",
"password").withLoadBalancingPolicy(DCAwareRoundRobinPolicy.builder().build())
.withSocketOptions(new
SocketOptions().setConnectTimeoutMillis(800).setKeepAlive(true)).build();
try {
mySession = myCluster.connect();
}
catch(Exception e) {
e.printStackTrace();
}
System.out.println("Session Established");
}


 private SSLOptions getSSLOption()
{
InputStream trustStore = null;
try
{
String theTrustStorePath =
"/var/opt/SecureInterface/myTrustStore.jks";
String theTrustStorePassword = "mypassword";
List theCipherSuites = new ArrayList();
theCipherSuites.add("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384");
KeyStore ks = KeyStore.getInstance("JKS");
*trustStore = new FileInputStream(theTrustStorePath);*
ks.load(trustStore, theTrustStorePassword.toCharArray());
TrustManagerFactory tmf =
TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init(ks);
SslContextBuilder builder =
SslContextBuilder.forClient()
.sslProvider(SslProvider.OPENSSL)
.trustManager(tmf)
.ciphers(theCipherSuites)
// only if you use client authentication
.keyManager(new
File("/var/opt/SecureInterface/keystore/Cass.crt"),
new
File("/var/opt/vs/SecureInterface/keystore/Cass_enc.key"));
SSLOptions sslOptions = new NettySSLOptions(builder.build());
return sslOptions;
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
try
{
trustStore.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
return null;
}

Cassandra server is running fine with client and server encryption options.
Moreover  I am able to run my client using JdkSSLOptions but have problem
with NettySSLOptions.

Has anyone implemented the  NettySSLOptions for Cassandra client
application?


Regards,
Jahar Tyagi

Re: saving distinct data in cassandra result in many tombstones

[shalom sagges]

 1. How to use sharding partition key in a way that partitions end up in
different nodes?
You could, for example, create a table with a bucket column added to the
partition key:
Table distinct(
hourNumber int,
bucket int, //could be a 5 minute bucket for example
key text,
distinctValue long
primary key ((hourNumber,bucket))
)

2. if i set gc_grace_seconds to 0, would it replace the row at memtable
(not saving repeated rows in sstables) or it would be done at first
compaction?
Overlapping rows in the memtables are merged regardless of the
gc_grace_seconds period. Setting gc_grace_seconds to 0 will immediately
evict tombstones during compaction but will disable hints delivery. You
should set gc_grace_seconds>max_hint_window_in_ms



On Tue, Jun 19, 2018 at 7:23 AM, onmstester onmstester 
wrote:

> Two other questions:
> 1. How to use sharding partition key in a way that partitions end up in
> different nodes?
> 2. if i set gc_grace_seconds to 0, would it replace the row at memtable
> (not saving repeated rows in sstables) or it would be done at first
> compaction?
>
> Sent using Zoho Mail 
>
>
>  On Tue, 19 Jun 2018 08:16:28 +0430 *onmstester onmstester
> >* wrote 
>
> Can i set gc_grace_seconds to 0 in this case? because reappearing deleted
> data has no impact on my Business Logic, i'm just either creating a new row
> or replacing the exactly same row.
>
> Sent using Zoho Mail 
>
>
>  On Wed, 13 Jun 2018 03:41:51 +0430 *Elliott Sims
> >* wrote 
>
>
>
> If this is data that expires after a certain amount of time, you probably
> want to look into using TWCS and TTLs to minimize the number of tombstones.
> Decreasing gc_grace_seconds then compacting will reduce the number of
> tombstones, but at the cost of potentially resurrecting deleted data if the
> table hasn't been repaired during the grace interval.  You can also just
> increase the tombstone thresholds, but the queries will be pretty
> expensive/wasteful.
>
> On Tue, Jun 12, 2018 at 2:02 AM, onmstester onmstester <
> onmstes...@zoho.com> wrote:
>
>
> Hi,
>
> I needed to save a distinct value for a key in each hour, the problem with
> saving everything and computing distincts in memory is that there
> are too many repeated data.
> Table schema:
> Table distinct(
> hourNumber int,
> key text,
> distinctValue long
> primary key (hourNumber)
> )
>
> I want to retrieve distinct count of all keys in a specific hour and using
> this data model it would be achieved by reading a single partition.
> The problem : i can't read from this table, system.log indicates that more
> than 100K tombstones read and no live data in it. The gc_grace time is
> the default (10 days), so i thought decreasing it to 1 hour and run
> compaction, but is this a right approach at all? i mean the whole idea of
> replacing
> some millions of rows. each  10 times in a partition again and again that
> creates alot of tombstones just to achieve distinct behavior?
>
> Thanks in advance
>
> Sent using Zoho Mail 
>
>
>
>

Tombstone

[Abhishek Singh]

Hi all,
   We using Cassandra for storing events which are time series
based for batch processing once a particular batch based on hour is
processed we delete the entries but we were left with almost 18% deletes
marked as Tombstones.
 I ran compaction on the particular CF tombstone didn't
come down.
Can anyone suggest what is the optimal tunning/recommended
practice used for compaction strategy and GC_grace period with 100k entries
and deletes every hour.

Warm Regards
Abhishek Singh

[no subject]

[Deniz Acay]

Hello there,

Let me get straight to the point. Yesterday our three node Cassandra
production cluster had a problem and we could not find a solution yet.
Before taking more radical actions, I would like to consult you about the
issue.

We are using Cassandra version 3.11.0. Cluster is living on AWS EC2 nodes
of type m4.2xlarge with 32 GBs of RAM. Each node Dockerized using host
networking mode. Two EBS SSD volumes are attached to each node, 32GB for
commit logs (io1) and 4TB for data directory (gp2). We have been running
smoothly for 7 months and filled %55 of data directory on each node.
Now our C* nodes fail during bootstrapping phase. Let me paste the logs
from system.log file from start to the time of error:

INFO  [main] 2018-06-19 09:51:32,726 YamlConfigurationLoader.java:89 -
Configuration location:
file:/opt/apache-cassandra-3.11.0/conf/cassandra.yaml
INFO  [main] 2018-06-19 09:51:32,954 Config.java:481 - Node
configuration:[allocate_tokens_for_keyspace=botanalytics;
authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer;
auto_bootstrap=false; auto_snapshot=true; back_pressure_enabled=false;
back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9,
factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50;
batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
broadcast_address=null; broadcast_rpc_address=null;
buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000;
cdc_enabled=false; cdc_free_space_check_interval_ms=250;
cdc_raw_directory=/var/data/cassandra/cdc_raw; cdc_total_space_in_mb=0;
client_encryption_options=; cluster_name=Botanalytics Production;
column_index_cache_size_in_kb=2; column_index_size_in_kb=64;
commit_failure_policy=stop_commit; commitlog_compression=null;
commitlog_directory=/var/data/cassandra_commitlog;
commitlog_max_compression_buffers_in_pool=3;
commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32;
commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN;
commitlog_sync_period_in_ms=1; commitlog_total_space_in_mb=8192;
compaction_large_partition_warning_threshold_mb=100;
compaction_throughput_mb_per_sec=1600; concurrent_compactors=null;
concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=64;
counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1;
credentials_validity_in_ms=2000; cross_node_timeout=false;
data_file_directories=[Ljava.lang.String;@662b4c69; disk_access_mode=auto;
disk_failure_policy=best_effort;
disk_optimization_estimate_percentile=0.95;
disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd;
dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1;
dynamic_snitch_reset_interval_in_ms=60;
dynamic_snitch_update_interval_in_ms=100;
enable_scripted_user_defined_functions=false;
enable_user_defined_functions=false;
enable_user_defined_functions_threads=true; encryption_options=null;
endpoint_snitch=Ec2Snitch; file_cache_size_in_mb=null;
gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000;
hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true;
hinted_handoff_throttle_in_kb=1024; hints_compression=null;
hints_directory=null; hints_flush_period_in_ms=1;
incremental_backups=false; index_interval=null;
index_summary_capacity_in_mb=null;
index_summary_resize_interval_in_minutes=60; initial_token=null;
inter_dc_stream_throughput_outbound_megabits_per_sec=200;
inter_dc_tcp_nodelay=false; internode_authenticator=null;
internode_compression=dc; internode_recv_buff_size_in_bytes=0;
internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647;
key_cache_save_period=14400; key_cache_size_in_mb=null;
listen_address=172.31.6.233; listen_interface=null;
listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false;
max_hint_window_in_ms=1080; max_hints_delivery_threads=2;
max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null;
max_streaming_retries=3; max_value_size_in_mb=256;
memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null;
memtable_flush_writers=0; memtable_heap_space_in_mb=null;
memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50;
native_transport_max_concurrent_connections=-1;
native_transport_max_concurrent_connections_per_ip=-1;
native_transport_max_frame_size_in_mb=256;
native_transport_max_threads=128; native_transport_port=9042;
native_transport_port_ssl=null; num_tokens=8;
otc_backlog_expiration_interval_ms=200;
otc_coalescing_enough_coalesced_messages=8;
otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200;
partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1;
permissions_validity_in_ms=2000; phi_convict_threshold=8.0;
prepared_st

Re:

[Vsevolod Filaretov]

Kurt, thank you very much for your answer! Your remark on GC totally
changed my thoughts on cassandra resources usage.

So.. more questions to the respective audience underway.

What is generally considered as

1) "too large" page size,
2)"large" page size
3) "normal conditions" page size?

How exactly fetch size affects CPU? Can too large page size provoke severe
CPU usage for constant GC, thus affecting Cassandra performance on read
requests (because CPU basically doesn't work on other tasks, while it's
constantly GCing)?

Thank you all very much!

пн, 18 июн. 2018 г., 14:28 kurt greaves :

> 1) Am I correct to assume that the larger page size some user session has
>> set - the larger portion of cluster/coordinator node resources will be
>> hogged by the corresponding session?
>> 2) Do I understand correctly that page size (imagine we have no timeout
>> settings) is limited by RAM and iops which I want to hand down to a single
>> user session?
>
> Yes for both of the above. More rows will be pulled into memory
> simultaneously with a larger page size, thus using more memory and IO.
>
> 3) Am I correct to assume that the page size/read request timeout
>> allowance I set is direct representation of chance to lock some node to
>> single user's requests?
>
> Concurrent reads can occur on a node, so it shouldn't "lock" the node to a
> single users request. However you can overload the node, which may be
> effectively the same thing. Don't set page sizes too high, otherwise the
> coordinator of the query will end up doing a lot of GC.
>
>
>