Re: Log output when Cassandra is "up"?
I have always relied upon the "Starting listening for CQL clients" message as a solid indicator of service readiness. John On Wed, Jan 8, 2020 at 3:37 PM Voytek Jarnot wrote: > Needing to know when Cassandra is finished initializing and is up & > running. > > Had some scripts which were looking through system.log for "No gossip > backlog; proceeding", but that turns out not to be 100% reliable. > > Is looking for "Starting listening for CQL clients" considered definitive? > I.E., always gets output on success, and not on failure? > > Thanks >
Re: Data is not syncing up when we add one more Node(DR) to existing 3 node cluster
Hi Anil,
In the cassandra.yaml file on your new node in DC2, is the IP address for
the seeds set to the seed node in DC1?
Best,
John
On Wed, Dec 11, 2019 at 11:09 PM Anil Kumar Ganipineni <
akganipin...@adaequare.com> wrote:
> Hi All,
>
>
>
> We have 3 node cluster on datacentre DC1 and below is our key space
> declaration. The current data size on the cluster is ~10GB. When we add a
> new node on datacentre DC2, the new node is not syncing up with the data,
> but it is showing UN when I run the *nodetool status*.
>
>
>
> *CREATE* KEYSPACE *production* *WITH* REPLICATION = { 'class' :
> 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'DC1': '3', 'DC2':
> '1' } *AND* DURABLE_WRITES = *true*;
>
>
>
>
>
> Please provide suggestions to make the new node on DC2 to sync up with
> existing cluster. This is required as the DC2 is our DR in a different
> region other than existing cluster.
>
>
>
>
>
> *Regards,*
>
> *Anil Ganipineni*
>
>
>
> *P** Please consider environment before printing this page.*
>
>
>
Re: "Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB"
Reid, I've only been working with Cassandra for 2 years, and this echoes my experience as well. Regarding the cache use, I know every use case is different, but have you experimented and found any performance benefit to increasing its size? Thanks, John Belliveau On Mon, Dec 2, 2019, 11:07 AM Reid Pinchback wrote: > Rahul, if my memory of this is correct, that particular logging message is > noisy, the cache is pretty much always used to its limit (and why not, it’s > a cache, no point in using less than you have). > > > > No matter what value you set, you’ll just change the “reached (….)” part > of it. I think what would help you more is to work with the team(s) that > have apps depending upon C* and decide what your performance SLA is with > them. If you are meeting your SLA, you don’t care about noisy messages. > If you aren’t meeting your SLA, then the noisy messages become sources of > ideas to look at. > > > > One thing you’ll find out pretty quickly. There are a lot of knobs you > can turn with C*, too many to allow for easy answers on what you should > do. Figure out what your throughput and latency SLAs are, and you’ll know > when to stop tuning. Otherwise you’ll discover that it’s a rabbit hole you > can dive into and not come out of for weeks. > > > > > > *From: *Hossein Ghiyasi Mehr > *Reply-To: *"user@cassandra.apache.org" > *Date: *Monday, December 2, 2019 at 10:35 AM > *To: *"user@cassandra.apache.org" > *Subject: *Re: "Maximum memory usage reached (512.000MiB), cannot > allocate chunk of 1.000MiB" > > > > *Message from External Sender* > > It may be helpful: > https://thelastpickle.com/blog/2018/08/08/compression_performance.html > <https://urldefense.proofpoint.com/v2/url?u=https-3A__thelastpickle.com_blog_2018_08_08_compression-5Fperformance.html&d=DwMFaQ&c=9Hv6XPedRSA-5PSECC38X80c1h60_XWA4z1k_R1pROA&r=OIgB3poYhzp3_A7WgD7iBCnsJaYmspOa2okNpf6uqWc&m=BlMYluADfxjSCocEBuEfptXuOJCAamgGaQreoJcMRJQ&s=rPo3nouxhBU2Yf2HRb2Udl87roS0KkGuPr-l2ferKXA&e=> > > It's complex. Simple explanation, cassandra keeps sstables in memory based > on chunk size and sstable parts. It manage loading new sstables to memory > based on requests on different sstables correctly . You should be worry > about it (sstables loaded in memory) > > > *VafaTech.com - A Total Solution for Data Gathering & Analysis* > > > > > > On Mon, Dec 2, 2019 at 6:18 PM Rahul Reddy > wrote: > > Thanks Hossein, > > > > How does the chunks are moved out of memory (LRU?) if it want to make room > for new requests to get chunks?if it has mechanism to clear chunks from > cache what causes to cannot allocate chunk? Can you point me to any > documention? > > > > On Sun, Dec 1, 2019, 12:03 PM Hossein Ghiyasi Mehr > wrote: > > Chunks are part of sstables. When there is enough space in memory to cache > them, read performance will increase if application requests it again. > > > > Your real answer is application dependent. For example write heavy > applications are different than read heavy or read-write heavy. Real time > applications are different than time series data environments and ... . > > > > > > > > On Sun, Dec 1, 2019 at 7:09 PM Rahul Reddy > wrote: > > Hello, > > > > We are seeing memory usage reached 512 mb and cannot allocate 1MB. I see > this because file_cache_size_mb by default set to 512MB. > > > > Datastax document recommends to increase the file_cache_size. > > > > We have 32G over all memory allocated 16G to Cassandra. What is the > recommended value in my case. And also when does this memory gets filled up > frequent does nodeflush helps in avoiding this info messages? > >
RE: Constant blocking read repair for such a tiny table
Hi Patrick,
Is this time series data? If so, I have run into issues with repair on time
series data using the SizeTieredCompactionStrategy. I have had better luck
using the TimeWindowCompactionStrategy.
John
Sent from Mail for Windows 10
From: Patrick Lee
Sent: Thursday, October 3, 2019 5:14 PM
To: user@cassandra.apache.org
Subject: Constant blocking read repair for such a tiny table
I have a cluster that is running 3.11.4 ( was upgraded a while back from 2.1.16
). what I see is a steady rate of read repair which is about 10% constantly on
only this 1 table. Repairs have been run (actually several times). The table
does not have a lot of writes to it so after repair, or even after a read
repair I would expect it to be fine. the reason i'm having to dig into this so
much is for the fact that under a much large traffic load than their normal
traffic, latencies are higher than the app team wants
I mean this thing is tiny, it's a 12x12 cluster but this 1 table is like 1GB
per node on disk.
the application team is doing reads at LOCAL_QUORUM and I can simulate this on
that cluster by running a query using quorum and/or local_quorum and in the
trace can see every time running the query it comes back with a
DigestMismatchException no matter how many times I run it. that record hasn't
been updated by the application for several months.
repairs are scheduled and run every 7 days via reaper, recently in the past
week this table has been repaired at least 3 times. every time there are
mismatches and data streams back and forth but yet still a constant rate of
read repairs.
curious if anyone has any recommendations to look info further or have
experienced anything like this?
this node has been up for 24 hours.. this is the netstats for read repairs
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 7481
Mismatch (Blocking): 11425375
Mismatch (Background): 17
Pool Name Active Pending Completed Dropped
Large messages n/a 0 1232 0
Small messages n/a 0 395903678 0
Gossip messages n/a 0 603746 0
example of the schema... some modifications have been made to reduce
read_reapair and speculative_retry while troubleshooting..
CREATE TABLE keyspace.table1 (
item bigint,
price int,
start_date timestamp,
end_date timestamp,
created_date timestamp,
cost decimal,
list decimal,
item_id int,
modified_date timestamp,
status int,
PRIMARY KEY ((item, price), start_date, end_date)
) WITH CLUSTERING ORDER BY (start_date ASC, end_date ASC)
AND read_repair_chance = 0.0
AND dclocal_read_repair_chance = 0.0
AND gc_grace_seconds = 864000
AND bloom_filter_fp_chance = 0.01
AND caching = { 'keys' : 'ALL', 'rows_per_partition' : 'NONE' }
AND comment = ''
AND compaction = { 'class' :
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold' : 32, 'min_threshold' : 4 }
AND compression = { 'chunk_length_in_kb' : 4, 'class' :
'org.apache.cassandra.io.compress.LZ4Compressor' }
AND default_time_to_live = 0
AND speculative_retry = 'NONE'
AND min_index_interval = 128
AND max_index_interval = 2048
AND crc_check_chance = 1.0
AND cdc = false
AND memtable_flush_period_in_ms = 0;
