Re: Timeout for only one keyspace in cluster

[learner dba]

 Thanks Jonathan.
this is device addition use case where we can't assign same value to more than 
one device, so we need isolation property. With regular UUID, we may end up 
assigning same value to two devices. I will talk to dev team and see if this 
can be changed and handled at application level.
On Monday, July 23, 2018, 12:01:15 PM CDT, Jonathan Haddad 
 wrote:  
 
 You don’t get this guarantee with counters.  Do not use them for unique 
values. Use a UUID instead. 
On Mon, Jul 23, 2018 at 9:11 AM learner dba  
wrote:

 James,
Yes, counter is implemented due to valid reasons. We need this value column to  
have unique values being used at the time of registering new devices.On 
Monday, July 23, 2018, 10:07:54 AM CDT, James Shaw  wrote:  
 
 does your application really need counter ?  just an option.
Thanks,
JamesOn Mon, Jul 23, 2018 at 10:57 AM, learner dba  wrote:

 Thanks a lot Ben. This makes sense but feel bad that we don't have a solution 
yet. We can try consistency level one but that will be against general rule for 
having local_quorum for production. Also, consistency ONE will not guarantee 0 
race condition.
Is there any better solution?
 

  On Saturday, July 21, 2018, 8:27:57 PM CDT, Ben Slater 
 wrote:  
 
 

Note that that writetimeout exception can be C*s way of telling you when there 
is contention on a LWT (rather than actually timing out). See 
https://issues.apache.org/ jira/browse/CASSANDRA-9328
CheersBen


On Sun, 22 Jul 2018 at 11:20 Goutham reddy  wrote:



Hi,As it is a single partition key, try to update the key with only partition 
key instead of passing other columns. And try to set consistency level ONE. 
Cheers,Goutham.



On Fri, Jul 20, 2018 at 6:57 AM learner dba  
wrote:





 Anybody has any ideas about this? This is happening in production and we 
really need to fix it.
 





  On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba 
 wrote:  
 
 





 Our foreignid is unique idetifier and we did check for wide partitions; 
cfhistorgrams show all partitions are evenly sized:

Percentile  SSTables     Write Latency      Read Latency    Partition Size      
  Cell Count

                              (micros)          (micros)           (bytes)      
            

50%             0.00             29.52              0.00              1916      
          12

75%             0.00             42.51              0.00              2299      
          12

95%             0.00             61.21              0.00              2759      
          14

98%             0.00             73.46              0.00              2759      
          17

99%             0.00             88.15              0.00              2759      
          17

Min             0.00              9.89              0.00               150      
           2

Max             0.00             88.15              0.00           7007506      
       42510
any thing else that we can check?
 





  On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com 
 wrote:  
 
 





Your partition key is foreignid. You may have a large partition. Why not use 
foreignid+timebucket as partition key?







From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject: 
Timeout for only one keyspace in cluster







Hi,
We have a cluster with multiple keyspaces. All queries are performing good but 
write operation on few tables in one specific keyspace gets write timeout. 
Table has counter column and counter update query times out always. Any idea?









CREATE TABLE x.y (

    foreignid uuid,

    timebucket text,

    key text,

    timevalue int,

    value counter,

    PRIMARY KEY (foreignid, timebucket, key, timevalue)

) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)

    AND bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''









    AND compaction = {'class': 'org.apache.cassandra.db.compa 
ction.SizeTieredCompactionStra tegy', 'max_threshold': '32', 'min_threshold': 
'4'}









    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compr ess.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';









Query and Error:
UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key 
= ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
received from cassandra within timeout period
I verified CL=local_serialWe had been working on this issue for many days; any 
help will be much appreciated.







-- 
Regards
Goutham Reddy



-- 

Ben Slater
Chief Product Officer





Read our latest technical blog posts here.

This email h

Re: Timeout for only one keyspace in cluster

[Jonathan Haddad]

You don’t get this guarantee with counters.  Do not use them for unique
values. Use a UUID instead.

On Mon, Jul 23, 2018 at 9:11 AM learner dba 
wrote:

> James,
>
> Yes, counter is implemented due to valid reasons. We need this value
> column to  have unique values being used at the time of registering new
> devices.
> On Monday, July 23, 2018, 10:07:54 AM CDT, James Shaw 
> wrote:
>
>
> does your application really need counter ?  just an option.
>
> Thanks,
>
> James
> On Mon, Jul 23, 2018 at 10:57 AM, learner dba  invalid > wrote:
>
> Thanks a lot Ben. This makes sense but feel bad that we don't have a
> solution yet. We can try consistency level one but that will be against
> general rule for having local_quorum for production. Also, consistency ONE
> will not guarantee 0 race condition.
>
> Is there any better solution?
>
> On Saturday, July 21, 2018, 8:27:57 PM CDT, Ben Slater <
> ben.sla...@instaclustr.com> wrote:
>
>
> Note that that writetimeout exception can be C*s way of telling you when
> there is contention on a LWT (rather than actually timing out). See 
> https://issues.apache.org/
> jira/browse/CASSANDRA-9328
> 
>
> Cheers
> Ben
>
> On Sun, 22 Jul 2018 at 11:20 Goutham reddy 
> wrote:
>
> Hi,
> As it is a single partition key, try to update the key with only partition
> key instead of passing other columns. And try to set consistency level ONE.
>
> Cheers,
> Goutham.
>
> On Fri, Jul 20, 2018 at 6:57 AM learner dba  d> wrote:
>
> Anybody has any ideas about this? This is happening in production and we
> really need to fix it.
>
> On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba <
> cassandra...@yahoo.com.INVALI D> wrote:
>
>
> Our foreignid is unique idetifier and we did check for wide partitions;
> cfhistorgrams show all partitions are evenly sized:
>
> Percentile  SSTables Write Latency  Read LatencyPartition Size
>   Cell Count
>
>   (micros)  (micros)   (bytes)
>
>
> 50% 0.00 29.52  0.00  1916
>   12
>
> 75% 0.00 42.51  0.00  2299
>   12
>
> 95% 0.00 61.21  0.00  2759
>   14
>
> 98% 0.00 73.46  0.00  2759
>   17
>
> 99% 0.00 88.15  0.00  2759
>   17
>
> Min 0.00  9.89  0.00   150
> 2
>
> Max 0.00 88.15  0.00   7007506
> 42510
> any thing else that we can check?
>
> On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com <
> wxn...@zjqunshuo.com> wrote:
>
>
> Your partition key is foreignid. You may have a large partition. Why not
> use foreignid+timebucket as partition key?
>
> *From:* learner dba 
> *Date:* 2018-07-19 01:48
> *To:* User cassandra.apache.org 
> *Subject:* Timeout for only one keyspace in cluster
>
> Hi,
>
> We have a cluster with multiple keyspaces. All queries are performing good
> but write operation on few tables in one specific keyspace gets write
> timeout. Table has counter column and counter update query times out
> always. Any idea?
>
> CREATE TABLE x.y (
>
> foreignid uuid,
>
> timebucket text,
>
> key text,
>
> timevalue int,
>
> value counter,
>
> PRIMARY KEY (foreignid, timebucket, key, timevalue)
>
> ) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)
>
> AND bloom_filter_fp_chance = 0.01
>
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
> AND comment = ''
>
> AND compaction = {'class': 'org.apache.cassandra.db.compa
> ction.SizeTieredCompactionStra tegy', 'max_threshold': '32',
> 'min_threshold': '4'}
>
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compr ess.LZ4Compressor'}
>
> AND crc_check_chance = 1.0
>
> AND dclocal_read_repair_chance = 0.1
>
> AND default_time_to_live = 0
>
> AND gc_grace_seconds = 864000
>
> AND max_index_interval = 2048
>
> AND memtable_flush_period_in_ms = 0
>
> AND min_index_interval = 128
>
> AND read_repair_chance = 0.0
>
> AND speculative_retry = '99PERCENTILE';
>
> Query and Error:
>
> UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND 
> key = ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
> received from cassandra within timeout period
>
>
> I verified CL=local_serial
>
> We had been working on this issue for many days; any help will be much 
> appreciated.
>
>
>
> --
> Regards
> Goutham Reddy
>
> --
>
>
> *Ben Slater*
>
> *Chief Product Officer *
>
>    
>

Re: Timeout for only one keyspace in cluster

[learner dba]

 James,
Yes, counter is implemented due to valid reasons. We need this value column to  
have unique values being used at the time of registering new devices.On 
Monday, July 23, 2018, 10:07:54 AM CDT, James Shaw  wrote:  
 
 does your application really need counter ?  just an option.
Thanks,
James
On Mon, Jul 23, 2018 at 10:57 AM, learner dba  
wrote:

 Thanks a lot Ben. This makes sense but feel bad that we don't have a solution 
yet. We can try consistency level one but that will be against general rule for 
having local_quorum for production. Also, consistency ONE will not guarantee 0 
race condition.
Is there any better solution?
On Saturday, July 21, 2018, 8:27:57 PM CDT, Ben Slater 
 wrote:  
 
 Note that that writetimeout exception can be C*s way of telling you when there 
is contention on a LWT (rather than actually timing out). See 
https://issues.apache.org/ jira/browse/CASSANDRA-9328
CheersBen
On Sun, 22 Jul 2018 at 11:20 Goutham reddy  wrote:

Hi,As it is a single partition key, try to update the key with only partition 
key instead of passing other columns. And try to set consistency level ONE. 
Cheers,Goutham.
On Fri, Jul 20, 2018 at 6:57 AM learner dba  
wrote:

 Anybody has any ideas about this? This is happening in production and we 
really need to fix it.
On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba 
 wrote:  
 
  Our foreignid is unique idetifier and we did check for wide partitions; 
cfhistorgrams show all partitions are evenly sized:

Percentile  SSTables     Write Latency      Read Latency    Partition Size      
  Cell Count

                              (micros)          (micros)           (bytes)      
            

50%             0.00             29.52              0.00              1916      
          12

75%             0.00             42.51              0.00              2299      
          12

95%             0.00             61.21              0.00              2759      
          14

98%             0.00             73.46              0.00              2759      
          17

99%             0.00             88.15              0.00              2759      
          17

Min             0.00              9.89              0.00               150      
           2

Max             0.00             88.15              0.00           7007506      
       42510
any thing else that we can check?
On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com 
 wrote:  
 
 Your partition key is foreignid. You may have a large partition. Why not use 
foreignid+timebucket as partition key?

 From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject: 
Timeout for only one keyspace in clusterHi,
We have a cluster with multiple keyspaces. All queries are performing good but 
write operation on few tables in one specific keyspace gets write timeout. 
Table has counter column and counter update query times out always. Any idea?

CREATE TABLE x.y (

    foreignid uuid,

    timebucket text,

    key text,

    timevalue int,

    value counter,

    PRIMARY KEY (foreignid, timebucket, key, timevalue)

) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)

    AND bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''

    AND compaction = {'class': 'org.apache.cassandra.db.compa 
ction.SizeTieredCompactionStra tegy', 'max_threshold': '32', 'min_threshold': 
'4'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compr ess.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';

Query and Error:
UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key 
= ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
received from cassandra within timeout period
I verified CL=local_serialWe had been working on this issue for many days; any 
help will be much appreciated.



-- 
Regards
Goutham Reddy

-- 

Ben Slater
Chief Product Officer





Read our latest technical blog posts here.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and 
Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally privileged 
information.  If you are not the intended recipient, do not copy or disclose 
its content, but please reply to this email immediately and highlight the error 
to the sender and then immediately delete the message.
  

  

Re: Timeout for only one keyspace in cluster

[James Shaw]

does your application really need counter ?  just an option.

Thanks,

James

On Mon, Jul 23, 2018 at 10:57 AM, learner dba  wrote:

> Thanks a lot Ben. This makes sense but feel bad that we don't have a
> solution yet. We can try consistency level one but that will be against
> general rule for having local_quorum for production. Also, consistency ONE
> will not guarantee 0 race condition.
>
> Is there any better solution?
>
> On Saturday, July 21, 2018, 8:27:57 PM CDT, Ben Slater <
> ben.sla...@instaclustr.com> wrote:
>
>
> Note that that writetimeout exception can be C*s way of telling you when
> there is contention on a LWT (rather than actually timing out). See
> https://issues.apache.org/jira/browse/CASSANDRA-9328
>
> Cheers
> Ben
>
> On Sun, 22 Jul 2018 at 11:20 Goutham reddy 
> wrote:
>
> Hi,
> As it is a single partition key, try to update the key with only partition
> key instead of passing other columns. And try to set consistency level ONE.
>
> Cheers,
> Goutham.
>
> On Fri, Jul 20, 2018 at 6:57 AM learner dba 
> wrote:
>
> Anybody has any ideas about this? This is happening in production and we
> really need to fix it.
>
> On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba <
> cassandra...@yahoo.com.INVALID> wrote:
>
>
> Our foreignid is unique idetifier and we did check for wide partitions;
> cfhistorgrams show all partitions are evenly sized:
>
> Percentile  SSTables Write Latency  Read LatencyPartition Size
>   Cell Count
>
>   (micros)  (micros)   (bytes)
>
>
> 50% 0.00 29.52  0.00  1916
>   12
>
> 75% 0.00 42.51  0.00  2299
>   12
>
> 95% 0.00 61.21  0.00  2759
>   14
>
> 98% 0.00 73.46  0.00  2759
>   17
>
> 99% 0.00 88.15  0.00  2759
>   17
>
> Min 0.00  9.89  0.00   150
> 2
>
> Max 0.00 88.15  0.00   7007506
> 42510
> any thing else that we can check?
>
> On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com <
> wxn...@zjqunshuo.com> wrote:
>
>
> Your partition key is foreignid. You may have a large partition. Why not
> use foreignid+timebucket as partition key?
>
>
> *From:* learner dba 
> *Date:* 2018-07-19 01:48
> *To:* User cassandra.apache.org 
> *Subject:* Timeout for only one keyspace in cluster
> Hi,
>
> We have a cluster with multiple keyspaces. All queries are performing good
> but write operation on few tables in one specific keyspace gets write
> timeout. Table has counter column and counter update query times out
> always. Any idea?
>
> CREATE TABLE x.y (
>
> foreignid uuid,
>
> timebucket text,
>
> key text,
>
> timevalue int,
>
> value counter,
>
> PRIMARY KEY (foreignid, timebucket, key, timevalue)
>
> ) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)
>
> AND bloom_filter_fp_chance = 0.01
>
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
> AND comment = ''
>
> AND compaction = {'class': 'org.apache.cassandra.db.compa
> ction.SizeTieredCompactionStrategy', 'max_threshold': '32',
> 'min_threshold': '4'}
>
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
>
> AND crc_check_chance = 1.0
>
> AND dclocal_read_repair_chance = 0.1
>
> AND default_time_to_live = 0
>
> AND gc_grace_seconds = 864000
>
> AND max_index_interval = 2048
>
> AND memtable_flush_period_in_ms = 0
>
> AND min_index_interval = 128
>
> AND read_repair_chance = 0.0
>
> AND speculative_retry = '99PERCENTILE';
>
> Query and Error:
>
> UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND 
> key = ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
> received from cassandra within timeout period
>
>
> I verified CL=local_serial
>
> We had been working on this issue for many days; any help will be much 
> appreciated.
>
>
>
> --
> Regards
> Goutham Reddy
>
> --
>
>
> *Ben Slater*
>
> *Chief Product Officer *
>
>    
>
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender a

Re: Timeout for only one keyspace in cluster

[learner dba]

 Thanks a lot Ben. This makes sense but feel bad that we don't have a solution 
yet. We can try consistency level one but that will be against general rule for 
having local_quorum for production. Also, consistency ONE will not guarantee 0 
race condition.
Is there any better solution?
On Saturday, July 21, 2018, 8:27:57 PM CDT, Ben Slater 
 wrote:  
 
 Note that that writetimeout exception can be C*s way of telling you when there 
is contention on a LWT (rather than actually timing out). See 
https://issues.apache.org/jira/browse/CASSANDRA-9328
CheersBen
On Sun, 22 Jul 2018 at 11:20 Goutham reddy  wrote:

Hi,As it is a single partition key, try to update the key with only partition 
key instead of passing other columns. And try to set consistency level ONE. 
Cheers,Goutham.
On Fri, Jul 20, 2018 at 6:57 AM learner dba  
wrote:

 Anybody has any ideas about this? This is happening in production and we 
really need to fix it.
On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba 
 wrote:  
 
  Our foreignid is unique idetifier and we did check for wide partitions; 
cfhistorgrams show all partitions are evenly sized:

Percentile  SSTables     Write Latency      Read Latency    Partition Size      
  Cell Count

                              (micros)          (micros)           (bytes)      
            

50%             0.00             29.52              0.00              1916      
          12

75%             0.00             42.51              0.00              2299      
          12

95%             0.00             61.21              0.00              2759      
          14

98%             0.00             73.46              0.00              2759      
          17

99%             0.00             88.15              0.00              2759      
          17

Min             0.00              9.89              0.00               150      
           2

Max             0.00             88.15              0.00           7007506      
       42510
any thing else that we can check?
On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com 
 wrote:  
 
 Your partition key is foreignid. You may have a large partition. Why not use 
foreignid+timebucket as partition key?

 From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject: 
Timeout for only one keyspace in clusterHi,
We have a cluster with multiple keyspaces. All queries are performing good but 
write operation on few tables in one specific keyspace gets write timeout. 
Table has counter column and counter update query times out always. Any idea?

CREATE TABLE x.y (

    foreignid uuid,

    timebucket text,

    key text,

    timevalue int,

    value counter,

    PRIMARY KEY (foreignid, timebucket, key, timevalue)

) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)

    AND bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''

    AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';

Query and Error:
UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key 
= ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
received from cassandra within timeout period
I verified CL=local_serialWe had been working on this issue for many days; any 
help will be much appreciated.



-- 
Regards
Goutham Reddy

-- 

Ben Slater
Chief Product Officer





Read our latest technical blog posts here.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and 
Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally privileged 
information.  If you are not the intended recipient, do not copy or disclose 
its content, but please reply to this email immediately and highlight the error 
to the sender and then immediately delete the message.
  

Re: Timeout for only one keyspace in cluster

[learner dba]

 Goutham,
How will it make any difference? Could you please elaborate?
On Saturday, July 21, 2018, 8:20:31 PM CDT, Goutham reddy 
 wrote:  
 
 Hi,As it is a single partition key, try to update the key with only partition 
key instead of passing other columns. And try to set consistency level ONE. 
Cheers,Goutham.
On Fri, Jul 20, 2018 at 6:57 AM learner dba  
wrote:

 Anybody has any ideas about this? This is happening in production and we 
really need to fix it.
On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba 
 wrote:  
 
  Our foreignid is unique idetifier and we did check for wide partitions; 
cfhistorgrams show all partitions are evenly sized:

Percentile  SSTables     Write Latency      Read Latency    Partition Size      
  Cell Count

                              (micros)          (micros)           (bytes)      
            

50%             0.00             29.52              0.00              1916      
          12

75%             0.00             42.51              0.00              2299      
          12

95%             0.00             61.21              0.00              2759      
          14

98%             0.00             73.46              0.00              2759      
          17

99%             0.00             88.15              0.00              2759      
          17

Min             0.00              9.89              0.00               150      
           2

Max             0.00             88.15              0.00           7007506      
       42510
any thing else that we can check?
On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com 
 wrote:  
 
 Your partition key is foreignid. You may have a large partition. Why not use 
foreignid+timebucket as partition key?

 From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject: 
Timeout for only one keyspace in clusterHi,
We have a cluster with multiple keyspaces. All queries are performing good but 
write operation on few tables in one specific keyspace gets write timeout. 
Table has counter column and counter update query times out always. Any idea?

CREATE TABLE x.y (

    foreignid uuid,

    timebucket text,

    key text,

    timevalue int,

    value counter,

    PRIMARY KEY (foreignid, timebucket, key, timevalue)

) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)

    AND bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''

    AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';

Query and Error:
UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key 
= ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
received from cassandra within timeout period
I verified CL=local_serialWe had been working on this issue for many days; any 
help will be much appreciated.



-- 
Regards
Goutham Reddy
  

Re: Timeout for only one keyspace in cluster

[Ben Slater]

Note that that writetimeout exception can be C*s way of telling you when
there is contention on a LWT (rather than actually timing out). See
https://issues.apache.org/jira/browse/CASSANDRA-9328

Cheers
Ben

On Sun, 22 Jul 2018 at 11:20 Goutham reddy 
wrote:

> Hi,
> As it is a single partition key, try to update the key with only partition
> key instead of passing other columns. And try to set consistency level ONE.
>
> Cheers,
> Goutham.
>
> On Fri, Jul 20, 2018 at 6:57 AM learner dba 
> wrote:
>
>> Anybody has any ideas about this? This is happening in production and we
>> really need to fix it.
>>
>> On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba
>>  wrote:
>>
>>
>> Our foreignid is unique idetifier and we did check for wide partitions;
>> cfhistorgrams show all partitions are evenly sized:
>>
>> Percentile  SSTables Write Latency  Read LatencyPartition
>> SizeCell Count
>>
>>   (micros)  (micros)   (bytes)
>>
>>
>> 50% 0.00 29.52  0.00
>> 191612
>>
>> 75% 0.00 42.51  0.00
>> 229912
>>
>> 95% 0.00 61.21  0.00
>> 275914
>>
>> 98% 0.00 73.46  0.00
>> 275917
>>
>> 99% 0.00 88.15  0.00
>> 275917
>>
>> Min 0.00  9.89  0.00   150
>> 2
>>
>> Max 0.00 88.15  0.00   7007506
>> 42510
>> any thing else that we can check?
>>
>> On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com <
>> wxn...@zjqunshuo.com> wrote:
>>
>>
>> Your partition key is foreignid. You may have a large partition. Why not
>> use foreignid+timebucket as partition key?
>>
>>
>> *From:* learner dba 
>> *Date:* 2018-07-19 01:48
>> *To:* User cassandra.apache.org 
>> *Subject:* Timeout for only one keyspace in cluster
>> Hi,
>>
>> We have a cluster with multiple keyspaces. All queries are performing
>> good but write operation on few tables in one specific keyspace gets write
>> timeout. Table has counter column and counter update query times out
>> always. Any idea?
>>
>> CREATE TABLE x.y (
>>
>> foreignid uuid,
>>
>> timebucket text,
>>
>> key text,
>>
>> timevalue int,
>>
>> value counter,
>>
>> PRIMARY KEY (foreignid, timebucket, key, timevalue)
>>
>> ) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)
>>
>> AND bloom_filter_fp_chance = 0.01
>>
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>>
>> AND comment = ''
>>
>> AND compaction = {'class':
>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4'}
>>
>> AND compression = {'chunk_length_in_kb': '64', 'class':
>> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>>
>> AND crc_check_chance = 1.0
>>
>> AND dclocal_read_repair_chance = 0.1
>>
>> AND default_time_to_live = 0
>>
>> AND gc_grace_seconds = 864000
>>
>> AND max_index_interval = 2048
>>
>> AND memtable_flush_period_in_ms = 0
>>
>> AND min_index_interval = 128
>>
>> AND read_repair_chance = 0.0
>>
>> AND speculative_retry = '99PERCENTILE';
>>
>> Query and Error:
>>
>> UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND 
>> key = ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
>> received from cassandra within timeout period
>>
>>
>> I verified CL=local_serial
>>
>> We had been working on this issue for many days; any help will be much 
>> appreciated.
>>
>>
>>
>> --
> Regards
> Goutham Reddy
>
-- 


*Ben Slater*

*Chief Product Officer *

   


Read our latest technical blog posts here
.

This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Timeout for only one keyspace in cluster

[Goutham reddy]

Hi,
As it is a single partition key, try to update the key with only partition
key instead of passing other columns. And try to set consistency level ONE.

Cheers,
Goutham.

On Fri, Jul 20, 2018 at 6:57 AM learner dba 
wrote:

> Anybody has any ideas about this? This is happening in production and we
> really need to fix it.
>
> On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba
>  wrote:
>
>
> Our foreignid is unique idetifier and we did check for wide partitions;
> cfhistorgrams show all partitions are evenly sized:
>
> Percentile  SSTables Write Latency  Read LatencyPartition Size
>   Cell Count
>
>   (micros)  (micros)   (bytes)
>
>
> 50% 0.00 29.52  0.00  1916
>   12
>
> 75% 0.00 42.51  0.00  2299
>   12
>
> 95% 0.00 61.21  0.00  2759
>   14
>
> 98% 0.00 73.46  0.00  2759
>   17
>
> 99% 0.00 88.15  0.00  2759
>   17
>
> Min 0.00  9.89  0.00   150
> 2
>
> Max 0.00 88.15  0.00   7007506
> 42510
> any thing else that we can check?
>
> On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com <
> wxn...@zjqunshuo.com> wrote:
>
>
> Your partition key is foreignid. You may have a large partition. Why not
> use foreignid+timebucket as partition key?
>
>
> *From:* learner dba 
> *Date:* 2018-07-19 01:48
> *To:* User cassandra.apache.org 
> *Subject:* Timeout for only one keyspace in cluster
> Hi,
>
> We have a cluster with multiple keyspaces. All queries are performing good
> but write operation on few tables in one specific keyspace gets write
> timeout. Table has counter column and counter update query times out
> always. Any idea?
>
> CREATE TABLE x.y (
>
> foreignid uuid,
>
> timebucket text,
>
> key text,
>
> timevalue int,
>
> value counter,
>
> PRIMARY KEY (foreignid, timebucket, key, timevalue)
>
> ) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)
>
> AND bloom_filter_fp_chance = 0.01
>
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>
> AND comment = ''
>
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
>
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
>
> AND crc_check_chance = 1.0
>
> AND dclocal_read_repair_chance = 0.1
>
> AND default_time_to_live = 0
>
> AND gc_grace_seconds = 864000
>
> AND max_index_interval = 2048
>
> AND memtable_flush_period_in_ms = 0
>
> AND min_index_interval = 128
>
> AND read_repair_chance = 0.0
>
> AND speculative_retry = '99PERCENTILE';
>
> Query and Error:
>
> UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND 
> key = ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
> received from cassandra within timeout period
>
>
> I verified CL=local_serial
>
> We had been working on this issue for many days; any help will be much 
> appreciated.
>
>
>
> --
Regards
Goutham Reddy

Re: Timeout for only one keyspace in cluster

[learner dba]

 Anybody has any ideas about this? This is happening in production and we 
really need to fix it.
On Thursday, July 19, 2018, 10:41:59 AM CDT, learner dba 
 wrote:  
 
  Our foreignid is unique idetifier and we did check for wide partitions; 
cfhistorgrams show all partitions are evenly sized:

Percentile  SSTables     Write Latency      Read Latency    Partition Size      
  Cell Count

                              (micros)          (micros)           (bytes)      
            

50%             0.00             29.52              0.00              1916      
          12

75%             0.00             42.51              0.00              2299      
          12

95%             0.00             61.21              0.00              2759      
          14

98%             0.00             73.46              0.00              2759      
          17

99%             0.00             88.15              0.00              2759      
          17

Min             0.00              9.89              0.00               150      
           2

Max             0.00             88.15              0.00           7007506      
       42510
any thing else that we can check?
On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com 
 wrote:  
 
 #yiv4986088840 body {line-height:1.5;}#yiv4986088840 blockquote 
{margin-top:0px;margin-bottom:0px;margin-left:0.5em;}#yiv4986088840 p 
{margin-top:0px;margin-bottom:0px;}#yiv4986088840 
div.yiv4986088840foxdiv20180719114102755131 {}#yiv4986088840 body 
{font-size:10.5pt;color:rgb(0, 0, 0);line-height:1.5;}Your partition key is 
foreignid. You may have a large partition. Why not use foreignid+timebucket as 
partition key?

 From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject: 
Timeout for only one keyspace in clusterHi,
We have a cluster with multiple keyspaces. All queries are performing good but 
write operation on few tables in one specific keyspace gets write timeout. 
Table has counter column and counter update query times out always. Any idea?

CREATE TABLE x.y (

    foreignid uuid,

    timebucket text,

    key text,

    timevalue int,

    value counter,

    PRIMARY KEY (foreignid, timebucket, key, timevalue)

) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)

    AND bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''

    AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';

Query and Error:
UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key 
= ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
received from cassandra within timeout period
I verified CL=local_serialWe had been working on this issue for many days; any 
help will be much appreciated.

Re: Timeout for only one keyspace in cluster

[learner dba]

 Our foreignid is unique idetifier and we did check for wide partitions; 
cfhistorgrams show all partitions are evenly sized:

Percentile  SSTables     Write Latency      Read Latency    Partition Size      
  Cell Count

                              (micros)          (micros)           (bytes)      
            

50%             0.00             29.52              0.00              1916      
          12

75%             0.00             42.51              0.00              2299      
          12

95%             0.00             61.21              0.00              2759      
          14

98%             0.00             73.46              0.00              2759      
          17

99%             0.00             88.15              0.00              2759      
          17

Min             0.00              9.89              0.00               150      
           2

Max             0.00             88.15              0.00           7007506      
       42510
any thing else that we can check?
On Wednesday, July 18, 2018, 10:44:29 PM CDT, wxn...@zjqunshuo.com 
 wrote:  
 
 #yiv9106249046 body {line-height:1.5;}#yiv9106249046 blockquote 
{margin-top:0px;margin-bottom:0px;margin-left:0.5em;}#yiv9106249046 p 
{margin-top:0px;margin-bottom:0px;}#yiv9106249046 
div.yiv9106249046foxdiv20180719114102755131 {}#yiv9106249046 body 
{font-size:10.5pt;color:rgb(0, 0, 0);line-height:1.5;}Your partition key is 
foreignid. You may have a large partition. Why not use foreignid+timebucket as 
partition key?

 From: learner dbaDate: 2018-07-19 01:48To: User cassandra.apache.orgSubject: 
Timeout for only one keyspace in clusterHi,
We have a cluster with multiple keyspaces. All queries are performing good but 
write operation on few tables in one specific keyspace gets write timeout. 
Table has counter column and counter update query times out always. Any idea?

CREATE TABLE x.y (

    foreignid uuid,

    timebucket text,

    key text,

    timevalue int,

    value counter,

    PRIMARY KEY (foreignid, timebucket, key, timevalue)

) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)

    AND bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''

    AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';

Query and Error:
UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key 
= ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
received from cassandra within timeout period
I verified CL=local_serialWe had been working on this issue for many days; any 
help will be much appreciated.


  

Re: Timeout for only one keyspace in cluster

[wxn...@zjqunshuo.com]

Your partition key is foreignid. You may have a large partition. Why not use 
foreignid+timebucket as partition key?
 
From: learner dba
Date: 2018-07-19 01:48
To: User cassandra.apache.org
Subject: Timeout for only one keyspace in cluster
Hi,

We have a cluster with multiple keyspaces. All queries are performing good but 
write operation on few tables in one specific keyspace gets write timeout. 
Table has counter column and counter update query times out always. Any idea?

CREATE TABLE x.y (
foreignid uuid,
timebucket text,
key text,
timevalue int,
value counter,
PRIMARY KEY (foreignid, timebucket, key, timevalue)
) WITH CLUSTERING ORDER BY (timebucket ASC, key ASC, timevalue ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';

Query and Error:

UPDATE x.y SET value = value + 1 where foreignid = ? AND timebucket = ? AND key 
= ? AND timevalue = ?, err = &errors.errorString{s:\"gocql: no response 
received from cassandra within timeout period
I verified CL=local_serialWe had been working on this issue for many days; any 
help will be much appreciated.