Re: What causes NoHostAvailableException, WriteTimeoutException, and UnavailableException?
In our case, the timeouts were happening because internode authentication was turned on and by default the user column family in the system_auth keyspace is replicated only on 1 node. We also had to tune the permissions_validity_in_ms from the default of 2000 ms to a larger value. The issue was that all authentication requests would go to one node, since it was replicated only on 1 node. We set replication factor to n (# of nodes) on the system_auth keyspace. Hope this helps. Parag From: Robert Coli rc...@eventbrite.commailto:rc...@eventbrite.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, November 24, 2014 at 2:52 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: What causes NoHostAvailableException, WriteTimeoutException, and UnavailableException? On Mon, Nov 24, 2014 at 12:57 PM, Kevin Burton bur...@spinn3r.commailto:bur...@spinn3r.com wrote: I’m trying to track down some exceptions in our production cluster. I bumped up our write load and now I’m getting a non-trivial number of these exceptions. Somewhere on the order of 100 per hour. All machines have a somewhat high CPU load because they’re doing other tasks. I’m worried that perhaps my background tasks are just overloading cassandra and one way to mitigate this is to nice them to least favorable priority (this is my first tasks). Two out of three of them are timeouts or lack of availability. Seeing this across your cluster is usually associated with hitting a pre-fail condition in terms of GC, where the amount of data stored per node makes the steady state working set larger than available non-fragmented heap. If you're graphing GC time, I would expect to see a concomitant spike there. =Rob
Re: Disabling compaction
Cassandra Version: 2.0.7
In my application, I am using Cassandra Java Driver 2.0.2
Thanks
Parag
From: Marcus Eriksson krum...@gmail.commailto:krum...@gmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Thursday, October 9, 2014 at 11:56 PM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: Disabling compaction
what version are you on?
On Thu, Oct 9, 2014 at 10:33 PM, Parag Shah
ps...@proofpoint.commailto:ps...@proofpoint.com wrote:
Hi all,
I am trying to disable compaction for a few select tables. Here is a
definition of one such table:
CREATE TABLE blob_2014_12_31 (
blob_id uuid,
blob_index int,
blob_chunk blob,
PRIMARY KEY (blob_id, blob_index)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'enabled': 'false', 'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor’};
I have set compaction ‘enabled’ : ‘false’ on the above table.
However, I do see compactions being run for this node:
-bash-3.2$ nodetool compactionstats
pending tasks: 55
compaction typekeyspace table completed
total unit progress
Compaction ids_high_awslab blob_2014_11_15
18122816990tel:18122816990 35814893020 bytes50.60%
Compaction ids_high_awslab blob_2014_12_31
18576750966tel:18576750966 34242866468 bytes54.25%
Compaction ids_high_awslab blob_2014_12_15 19213914904
35956698600 bytes53.44%
Active compaction remaining time : 0h49m46s
Can you someone tell me why this is happening? Do I need to set the compaction
threshold to 0 0?
Regards
Parag
Re: Disabling compaction
Thanks, Marcus.
Regards
Parag
From: Marcus Eriksson krum...@gmail.commailto:krum...@gmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Friday, October 10, 2014 at 10:59 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: Disabling compaction
this is fixed in 2.0.8;
https://issues.apache.org/jira/browse/CASSANDRA-7187https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D7187d=AAMFaQc=Vxt5e0Osvvt2gflwSlsJ5DmPGcPvTRKLJyp031rXjhgr=Zm-paw4L-LBldtJybEWfBLLjvC5axjko2vu-eqkngMIm=7buRq9Dt25OaWjoEDHVcOqu1FQ4kYiNX3BLqnGNXYXIs=cGXvzXOISsNR4t-L5pm837TZYO5fk4uRra8MzdoD7Use=
/Marcus
On Fri, Oct 10, 2014 at 7:11 PM, Parag Shah
ps...@proofpoint.commailto:ps...@proofpoint.com wrote:
Cassandra Version: 2.0.7
In my application, I am using Cassandra Java Driver 2.0.2
Thanks
Parag
From: Marcus Eriksson krum...@gmail.commailto:krum...@gmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Thursday, October 9, 2014 at 11:56 PM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: Disabling compaction
what version are you on?
On Thu, Oct 9, 2014 at 10:33 PM, Parag Shah
ps...@proofpoint.commailto:ps...@proofpoint.com wrote:
Hi all,
I am trying to disable compaction for a few select tables. Here is a
definition of one such table:
CREATE TABLE blob_2014_12_31 (
blob_id uuid,
blob_index int,
blob_chunk blob,
PRIMARY KEY (blob_id, blob_index)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'enabled': 'false', 'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor’};
I have set compaction ‘enabled’ : ‘false’ on the above table.
However, I do see compactions being run for this node:
-bash-3.2$ nodetool compactionstats
pending tasks: 55
compaction typekeyspace table completed
total unit progress
Compaction ids_high_awslab blob_2014_11_15
18122816990tel:18122816990 35814893020 bytes50.60%
Compaction ids_high_awslab blob_2014_12_31
18576750966tel:18576750966 34242866468 bytes54.25%
Compaction ids_high_awslab blob_2014_12_15 19213914904
35956698600 bytes53.44%
Active compaction remaining time : 0h49m46s
Can you someone tell me why this is happening? Do I need to set the compaction
threshold to 0 0?
Regards
Parag
Disabling compaction
Hi all,
I am trying to disable compaction for a few select tables. Here is a
definition of one such table:
CREATE TABLE blob_2014_12_31 (
blob_id uuid,
blob_index int,
blob_chunk blob,
PRIMARY KEY (blob_id, blob_index)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'enabled': 'false', 'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor’};
I have set compaction ‘enabled’ : ‘false’ on the above table.
However, I do see compactions being run for this node:
-bash-3.2$ nodetool compactionstats
pending tasks: 55
compaction typekeyspace table completed
total unit progress
Compaction ids_high_awslab blob_2014_11_15 18122816990
35814893020 bytes50.60%
Compaction ids_high_awslab blob_2014_12_31 18576750966
34242866468 bytes54.25%
Compaction ids_high_awslab blob_2014_12_15 19213914904
35956698600 bytes53.44%
Active compaction remaining time : 0h49m46s
Can you someone tell me why this is happening? Do I need to set the compaction
threshold to 0 0?
Regards
Parag
