Re: READ Queries timing out.

[Jeff Jirsa]

On 2017-07-07 07:26 (-0700), Pranay akula  wrote: 
> Lately i am seeing some select queries timing out, data modelling to blame
> for but not in a situation to redo it.
> 
> Does increasing heap will help ??
> 
> currently using 1GB new_heap, I analysed the GC logs not having any issues
> with major GC's .
> 
> Using G1GC , does increasing new_heap will help ??
> 

Increasing new heap size generally helps when you're seeing a lot of promotion 
- if you're not seeing long major GCs, are you seeing a lot of promotion from 
eden to old gen? 

You don't typically set -Xmn (new heap size) when using G1GC 


> currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i
> increase heap to lets say 2GB is that effective b/c young GC's will kick in
> more frequently  to complete in 500ms right ??

Min heap size for G1 is probably 16G. You were saying "new_heap", now you're 
just saying heap - do you mean new here, or total heap size?


You tablestats show a max partition size of 75M, which isn't nearly as bad as I 
was expecting (or not nearly as bad as we often see when people ask this 
question). You do occasionally scan a lot of sstables (up to 20?), so I'm 
assuming you have STCS - you may benefit from switching to LCS to try to limit 
the number of sstables you touch on read. Also, if your data isn't in memory 
(if your data set is larger than RAM and reads are random), you may benefit 
from a much lower compression_chunk_size - the default is 64k, but 4k or 16k is 
often much better if you do have to read from disk.

- Jeff

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: private interface for interdc messaging

[CPC]

Thank you Nitan.

On Jul 7, 2017 8:59 PM, "Nitan Kainth"  wrote:

Yes. Because that's the ip used for internode communication

Sent from my iPhone

On Jul 7, 2017, at 10:52 AM, CPC  wrote:

Hi Nitan,

Do you mean setting broadcast_address to private network would suffice?

On 7 July 2017 at 20:45, Nitan Kainth  wrote:

> You can setup broadcast address for the IP on which Nodes will communicate
> with each other. You network team can setup routing table from source to
> target.
> We had similar setup done in one of my previous project where we
> segregated network between application and C* nodes communication.
>
> > On Jul 7, 2017, at 10:28 AM, CPC  wrote:
> >
> > Hi,
> >
> > We are building 2 datacenters with each machine have one public(for
> native client connections) and one for private(internode communication).
> What we noticed that nodes in one datacenters trying to communicate with
> other nodes in other dc over their public interfaces.
> > I mean:
> > DC1 Node1 public interface -> DC2 Node1 private interface
> > But what we perefer is:
> > DC1 Node1 private interface -> DC2 Node1 private interface
> >
> > Is there any configuration so a node make interdc connection over its
> private network?
> >
> > Thank you...
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: private interface for interdc messaging

[Nitan Kainth]

Yes. Because that's the ip used for internode communication

Sent from my iPhone

> On Jul 7, 2017, at 10:52 AM, CPC  wrote:
> 
> Hi Nitan,
> 
> Do you mean setting broadcast_address to private network would suffice? 
> 
>> On 7 July 2017 at 20:45, Nitan Kainth  wrote:
>> You can setup broadcast address for the IP on which Nodes will communicate 
>> with each other. You network team can setup routing table from source to 
>> target.
>> We had similar setup done in one of my previous project where we segregated 
>> network between application and C* nodes communication.
>> 
>> > On Jul 7, 2017, at 10:28 AM, CPC  wrote:
>> >
>> > Hi,
>> >
>> > We are building 2 datacenters with each machine have one public(for native 
>> > client connections) and one for private(internode communication). What we 
>> > noticed that nodes in one datacenters trying to communicate with other 
>> > nodes in other dc over their public interfaces.
>> > I mean:
>> > DC1 Node1 public interface -> DC2 Node1 private interface
>> > But what we perefer is:
>> > DC1 Node1 private interface -> DC2 Node1 private interface
>> >
>> > Is there any configuration so a node make interdc connection over its 
>> > private network?
>> >
>> > Thank you...
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 

Re: private interface for interdc messaging

[CPC]

Hi Nitan,

Do you mean setting broadcast_address to private network would suffice?

On 7 July 2017 at 20:45, Nitan Kainth  wrote:

> You can setup broadcast address for the IP on which Nodes will communicate
> with each other. You network team can setup routing table from source to
> target.
> We had similar setup done in one of my previous project where we
> segregated network between application and C* nodes communication.
>
> > On Jul 7, 2017, at 10:28 AM, CPC  wrote:
> >
> > Hi,
> >
> > We are building 2 datacenters with each machine have one public(for
> native client connections) and one for private(internode communication).
> What we noticed that nodes in one datacenters trying to communicate with
> other nodes in other dc over their public interfaces.
> > I mean:
> > DC1 Node1 public interface -> DC2 Node1 private interface
> > But what we perefer is:
> > DC1 Node1 private interface -> DC2 Node1 private interface
> >
> > Is there any configuration so a node make interdc connection over its
> private network?
> >
> > Thank you...
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: private interface for interdc messaging

[Nitan Kainth]

You can setup broadcast address for the IP on which Nodes will communicate with 
each other. You network team can setup routing table from source to target.
We had similar setup done in one of my previous project where we segregated 
network between application and C* nodes communication.

> On Jul 7, 2017, at 10:28 AM, CPC  wrote:
> 
> Hi,
> 
> We are building 2 datacenters with each machine have one public(for native 
> client connections) and one for private(internode communication). What we 
> noticed that nodes in one datacenters trying to communicate with other nodes 
> in other dc over their public interfaces. 
> I mean:
> DC1 Node1 public interface -> DC2 Node1 private interface
> But what we perefer is:
> DC1 Node1 private interface -> DC2 Node1 private interface
> 
> Is there any configuration so a node make interdc connection over its private 
> network?
> 
> Thank you...


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Understanding of cassandra metrics

[Chris Lohfink]

The coordinator read/scan (Scan is just different naming for the Range, so
coordinator view of RangeLatency) is the latencies from the coordinator
perspective, so it includes network latency between replicas and such. This
which is actually added for speculative retry (why there is no
coordinatorWriteLatency). Only the CoordinatorReadLatency is used for it
however.

The Read/RangeLatency metrics are for local reads, basically just how long
to read from disk and merge with sstables.

The View* metrics are only relevant to materialized views. There actually
is a partition lock for updates which ViewLockAcquireTime gives visibility
too. Also there are sometimes reads required for updating materialized
views, which ViewReadTime is for tracking. For more details id recommend
https://opencredo.com/everything-need-know-cassandra-materialized-views/

Chris

On Fri, Jul 7, 2017 at 9:42 AM, ZAIDI, ASAD A  wrote:

> What exactly does mean CoordinatorScanLatency for example
>
> CoordinatorScanLatency  is a timer metric that present coordinator range
> scan latency for  table.
>
> Is it latency on full table scan or maybe range scan by clustering key?
>
> It is range scan.. clustering key is used to only store
> data in sorted fashion – partition key along with chosen partitioner helps
> in range scan of data.
>
> Can anybody write into partition while locked?
>
> Writes are atomic – it depends on your chosen consistency
> level to determine if writes will fail or succeed.
>
>
>
> *From:* Павел Сапежко [mailto:amelius0...@gmail.com]
> *Sent:* Friday, July 07, 2017 8:23 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Understanding of cassandra metrics
>
>
>
> Are you really think that I don't read docs? Do you have enough
> information in the documentation? I think no. What exactly does mean 
> CoordinatorScanLatency
> for example? Is it latency on full table scan or maybe range scan by
> clustering key? What exactly mean ViewLockAcquireTime? What is "partition
> lock"? Can anybody write into partition while locked? Etc.
>
> пт, 7 июл. 2017 г. в 13:01, Ivan Iliev :
>
> 1st result on google returns:
>
>
>
> http://cassandra.apache.org/doc/latest/operating/metrics.html
> 
>
>
>
> On Fri, Jul 7, 2017 at 12:16 PM, Павел Сапежко 
> wrote:
>
> Hello, I have several question about cassandra metrics. What does exactly
> mean the next metrics:
>
>- CoordinatorReadLatency
>- CoordinatorScanLatency
>- ReadLatency
>- RangeLatency
>- ViewLockAcquireTime
>- ViewReadTime
>
> --
>
> С уважением,
>
> Павел Сапежко
>
> skype: p.sapezhko
>
>
>
> --
>
> С уважением,
>
> Павел Сапежко
>
> skype: p.sapezhko
>

private interface for interdc messaging

[CPC]

Hi,

We are building 2 datacenters with each machine have one public(for native
client connections) and one for private(internode communication). What we
noticed that nodes in one datacenters trying to communicate with other
nodes in other dc over their public interfaces.
I mean:
DC1 Node1 public interface -> DC2 Node1 private interface
But what we perefer is:
DC1 Node1 private interface -> DC2 Node1 private interface

Is there any configuration so a node make interdc connection over its
private network?

Thank you...

RE: READ Queries timing out.

[Durity, Sean R]

1 GB heap is very small. Why not try increasing it to 50% of RAM and see if it 
helps you track down the real issue. It is hard to tune around a bad data 
model, if that is indeed the issue. Seeing your tables and queries would help.


Sean Durity

From: Pranay akula [mailto:pranay.akula2...@gmail.com]
Sent: Friday, July 07, 2017 11:47 AM
To: user@cassandra.apache.org
Cc: ZAIDI, ASAD A 
Subject: Re: READ Queries timing out.

Thanks ZAIDI,

Using C++ driver doesn't have tracing with driver so executing those from 
cqlsh. when i am tracing i am getting below error, i increased 
--request-timeout to 3600 in cqlsh.

ReadTimeout: code=1200 [Coordinator node timed out waiting for replica nodes' 
responses] message="Operation timed out - received only 0 responses." 
info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
Statement trace did not complete within 10 seconds

The below are cfstats and cfhistograms, i can see  read latency, cell count and 
Maximum live cells per slice (last five minutes) are high. is there any way to 
get around this with out changing data model.

Percentile  SSTables Write Latency  Read Latency  Partition 
SizeCell Count
 (micros)  (micros) 
 (bytes)
50% 1.00 20.00   NaN
   133120
75% 2.00 29.00   NaN
  686686
95% 8.00 60.00   NaN
126934  1331
98%10.00103.00   NaN
   315852  3973
99%12.00149.00   NaN
  545791  8239
Min 0.00  0.000.00  
 104 0
Max20.00   12730764.00  9773372036884776000.00  
74975550 83457



Read Count: 44514407
Read Latency: 82.92876612928933 ms.
Write Count: 3007585812
Write Latency: 0.07094456590853208 ms.
Pending Flushes: 0
SSTable count: 9
Space used (live): 66946214374
Space used (total): 66946214374
Space used by snapshots (total): 0
Off heap memory used (total): 33706492
SSTable Compression Ratio: 0.5598380206656697
Number of keys (estimate): 2483819
Memtable cell count: 15008
Memtable data size: 330597
Memtable off heap memory used: 518502
Memtable switch count: 39915
Local read count: 44514407
Local read latency: 82.929 ms
Local write count: 3007585849
Local write latency: 0.071 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 12623632
Bloom filter off heap memory used: 12623560
Index summary off heap memory used: 3285614
Compression metadata off heap memory used: 17278816
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 74975550
Compacted partition mean bytes: 27111
Average live cells per slice (last five minutes): 
388.7486606077893
Maximum live cells per slice (last five minutes): 28983.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0


Thanks
Pranay.

On Fri, Jul 7, 2017 at 11:16 AM, Thakrar, Jayesh 
mailto:jthak...@conversantmedia.com>> wrote:
Can you provide more details.
E.g. table structure, the app used for the query, the query itself and the 
error message.

Also get the output of the following commands from your cluster nodes (note 
that one command uses "." and the other "space" between keyspace and tablename)

nodetool -h  tablestats .
nodetool -h  tablehistograms  

Timeouts can happen at the client/application level (which can be tuned) and at 
the coordinator node level (which too can be tuned).
But again those timeouts are a symptom of something.
It can happen at the client side because of connection pool queue too full 
(which is likely due to response time from the cluster/coordinate nodes).
And the issues at the cluster side could be due to several reasons.
E.g. your query has to scan through too many tombstones, causing the delay or 
your query (if using filter).

From: "ZAIDI, ASAD A" mailto:az1...@att.com>>
Date: Friday, July 7, 2017 at 9

Re: Starting Cassandrs after restore of Data - get error

[Nitan Kainth]

Yes it is possible, generally quotes gets messed up. I would suggest as 
alternate comment num_tokens and start. Then compare output of node tool ring 
on source and target.

> On Jul 7, 2017, at 9:47 AM, Jonathan Baynes  
> wrote:
> 
> Yes both clusters match I've checked 3 Times and diff'd it as well. Would 
> file format have any affect I'm amending on windows machine and returning the 
> file back to Linux 
> 
> Thanks
> J
> 
> Sent from my iPhone
> 
> On 7 Jul 2017, at 17:43, Nitan Kainth  > wrote:
> 
>> Jonathan,
>> 
>> Make sure initial tokens have values from back up cluster i.e. 256 tokens. 
>> It is possible to have typo.
>> 
>>> On Jul 7, 2017, at 9:14 AM, Jonathan Baynes >> > wrote:
>>> 
>>> Hi again,
>>>  
>>> Trying to restart my nodes after restoring snapshot data, initial tokens 
>>> have been added in as per the instructions online.
>>>  
>>> In system.log I get this error (same error is I run nodetool cleanup)
>>>  
>>> Exception encountered during startup: The number of initial tokens (by 
>>> initial_token) specified is different from num_tokens value
>>>  
>>>  
>>> On both Cluster A and Cluster B the Num_tokens = 256
>>>  
>>> Ive taken the initial tokens from running this script
>>>  
>>> nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | 
>>> awk '{print $NF ","}' | xargs > /tmp/tokens
>>>  
>>> when pasting in the tokens originally I got the an error, but this was due 
>>> to the spacing between the tokens. That error has been resolved I’m just 
>>> left with this one?
>>>  
>>> Any ideas
>>>  
>>> Thanks
>>> J
>>>  
>>> Jonathan Baynes
>>> DBA
>>> Tradeweb Europe Limited
>>> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
>>> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
>>> jonathan.bay...@tradeweb.com 
>>>  
>>>     follow us:   
>>> 
>>> 
>>> 
>>> —
>>> A leading marketplace  for 
>>> electronic fixed income, derivatives and ETF trading
>>>  
>>> 
>>> 
>>> This e-mail may contain confidential and/or privileged information. If you 
>>> are not the intended recipient (or have received this e-mail in error) 
>>> please notify the sender immediately and destroy it. Any unauthorized 
>>> copying, disclosure or distribution of the material in this e-mail is 
>>> strictly forbidden. Tradeweb reserves the right to monitor all e-mail 
>>> communications through its networks. If you do not wish to receive 
>>> marketing emails about our products / services, please let us know by 
>>> contacting us, either by email at contac...@tradeweb.com 
>>>  or by writing to us at the registered 
>>> office of Tradeweb in the UK, which is: Tradeweb Europe Limited (company 
>>> number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy 
>>> policy, visit our website @ www.tradeweb.com .
>>> 
>> 
> 
> 
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have received this e-mail in error) please 
> notify the sender immediately and destroy it. Any unauthorized copying, 
> disclosure or distribution of the material in this e-mail is strictly 
> forbidden. Tradeweb reserves the right to monitor all e-mail communications 
> through its networks. If you do not wish to receive marketing emails about 
> our products / services, please let us know by contacting us, either by email 
> at contac...@tradeweb.com or by writing to us at the registered office of 
> Tradeweb in the UK, which is: Tradeweb Europe Limited (company number 
> 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our privacy policy, 
> visit our website @ www.tradeweb.com.
> 

RE: Starting Cassandrs after restore of Data - get error

[Durity, Sean R]

I have seen Windows format cause problems. Run dos2unix on the cassandra.yaml 
file (on the linux box) and see if it helps.


Sean Durity
lord of the (C*) rings (Staff Systems Engineer - Cassandra)
MTC 2250
#cassandra - for the latest news and updates

From: Jonathan Baynes [mailto:jonathan.bay...@tradeweb.com]
Sent: Friday, July 07, 2017 12:48 PM
To: user@cassandra.apache.org
Subject: Re: Starting Cassandrs after restore of Data - get error

Yes both clusters match I've checked 3 Times and diff'd it as well. Would file 
format have any affect I'm amending on windows machine and returning the file 
back to Linux

Thanks
J

Sent from my iPhone

On 7 Jul 2017, at 17:43, Nitan Kainth 
mailto:ni...@bamlabs.com>> wrote:
Jonathan,

Make sure initial tokens have values from back up cluster i.e. 256 tokens. It 
is possible to have typo.

On Jul 7, 2017, at 9:14 AM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:

Hi again,

Trying to restart my nodes after restoring snapshot data, initial tokens have 
been added in as per the instructions online.

In system.log I get this error (same error is I run nodetool cleanup)

Exception encountered during startup: The number of initial tokens (by 
initial_token) specified is different from num_tokens value


On both Cluster A and Cluster B the Num_tokens = 256

Ive taken the initial tokens from running this script

nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk 
'{print $NF ","}' | xargs > /tmp/tokens

when pasting in the tokens originally I got the an error, but this was due to 
the spacing between the tokens. That error has been resolved I'm just left with 
this one?

Any ideas

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com


   follow us:  

   

-
A leading 
marketplace
 for electronic fixed income, derivatives and ETF trading


This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at 
the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited 
(company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our 
privacy policy, visit our website @ 
www.tradeweb.com.




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us

Re: Starting Cassandrs after restore of Data - get error

[Jonathan Baynes]

Yes both clusters match I've checked 3 Times and diff'd it as well. Would file 
format have any affect I'm amending on windows machine and returning the file 
back to Linux

Thanks
J

Sent from my iPhone

On 7 Jul 2017, at 17:43, Nitan Kainth 
mailto:ni...@bamlabs.com>> wrote:

Jonathan,

Make sure initial tokens have values from back up cluster i.e. 256 tokens. It 
is possible to have typo.

On Jul 7, 2017, at 9:14 AM, Jonathan Baynes 
mailto:jonathan.bay...@tradeweb.com>> wrote:

Hi again,

Trying to restart my nodes after restoring snapshot data, initial tokens have 
been added in as per the instructions online.

In system.log I get this error (same error is I run nodetool cleanup)

Exception encountered during startup: The number of initial tokens (by 
initial_token) specified is different from num_tokens value


On both Cluster A and Cluster B the Num_tokens = 256

Ive taken the initial tokens from running this script

nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk 
'{print $NF ","}' | xargs > /tmp/tokens

when pasting in the tokens originally I got the an error, but this was due to 
the spacing between the tokens. That error has been resolved I’m just left with 
this one?

Any ideas

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
jonathan.bay...@tradeweb.com

   follow us:  

   

—
A leading marketplace for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at 
the registered office of Tradeweb in the UK, which is: Tradeweb Europe Limited 
(company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To see our 
privacy policy, visit our website @ www.tradeweb.com.




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: Starting Cassandrs after restore of Data - get error

[Nitan Kainth]

Jonathan,

Make sure initial tokens have values from back up cluster i.e. 256 tokens. It 
is possible to have typo.

> On Jul 7, 2017, at 9:14 AM, Jonathan Baynes  
> wrote:
> 
> Hi again,
>  
> Trying to restart my nodes after restoring snapshot data, initial tokens have 
> been added in as per the instructions online.
>  
> In system.log I get this error (same error is I run nodetool cleanup)
>  
> Exception encountered during startup: The number of initial tokens (by 
> initial_token) specified is different from num_tokens value
>  
>  
> On both Cluster A and Cluster B the Num_tokens = 256
>  
> Ive taken the initial tokens from running this script
>  
> nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk 
> '{print $NF ","}' | xargs > /tmp/tokens
>  
> when pasting in the tokens originally I got the an error, but this was due to 
> the spacing between the tokens. That error has been resolved I’m just left 
> with this one?
>  
> Any ideas
>  
> Thanks
> J
>  
> Jonathan Baynes
> DBA
> Tradeweb Europe Limited
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988  •  F +44 (0)20 7776 3201  •  M +44 (0) xx
> jonathan.bay...@tradeweb.com 
>  
>     follow us:   
> 
> 
> —
> A leading marketplace  for 
> electronic fixed income, derivatives and ETF trading
>  
> 
> 
> This e-mail may contain confidential and/or privileged information. If you 
> are not the intended recipient (or have received this e-mail in error) please 
> notify the sender immediately and destroy it. Any unauthorized copying, 
> disclosure or distribution of the material in this e-mail is strictly 
> forbidden. Tradeweb reserves the right to monitor all e-mail communications 
> through its networks. If you do not wish to receive marketing emails about 
> our products / services, please let us know by contacting us, either by email 
> at contac...@tradeweb.com  or by writing to us 
> at the registered office of Tradeweb in the UK, which is: Tradeweb Europe 
> Limited (company number 3912826), 1 Fore Street Avenue London EC2Y 9DT. To 
> see our privacy policy, visit our website @ www.tradeweb.com 
> .
> 

RE: Repair on system_auth

[Mark Furlong]

I’m currently on 2.1.12. Are you saying this bug exists on the current latest 
version 3.0.14?

Thank you
Mark
801-705-7115 office

From: ­Fay Hou [Storage Service] [mailto:fay...@coupang.com]
Sent: Thursday, July 6, 2017 2:24 PM
To: User 
Subject: Re: Repair on system_auth

There is a bug on repair system_auth keyspace. We just skip the repair on 
system_auth. Yes. it is ok to kill the running repair job

On Thu, Jul 6, 2017 at 1:14 PM, Subroto Barua 
mailto:sbarua...@yahoo.com.invalid>> wrote:
you can check the status via nodetool netstats
to kill the repair job, restart the instance


On Thursday, July 6, 2017, 1:09:42 PM PDT, Mark Furlong 
mailto:mfurl...@ancestry.com>> wrote:



I have started a repair on my system_auth keyspace. The repair has started and 
the process shows as running with ps but am not seeing any CPU with top. I’m 
also note seeing any antientropy sessions building merkle trees in the log. Can 
I safely kill a repair and how?





Mark Furlong


Sr. Database Administrator


mfurl...@ancestry.com
M: 801-859-7427

O: 801-705-7115

1300 W Traverse Pkwy

Lehi, UT 84043








​[http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]







-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 
user-h...@cassandra.apache.org

Starting Cassandrs after restore of Data - get error

[Jonathan Baynes]

Hi again,

Trying to restart my nodes after restoring snapshot data, initial tokens have 
been added in as per the instructions online.

In system.log I get this error (same error is I run nodetool cleanup)

Exception encountered during startup: The number of initial tokens (by 
initial_token) specified is different from num_tokens value


On both Cluster A and Cluster B the Num_tokens = 256

Ive taken the initial tokens from running this script

nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk 
'{print $NF ","}' | xargs > /tmp/tokens

when pasting in the tokens originally I got the an error, but this was due to 
the spacing between the tokens. That error has been resolved I'm just left with 
this one?

Any ideas

Thanks
J

Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 

[cid:image003.jpg@01CD26AD.4165F110] 
-
A leading marketplace for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mail in error) please 
notify the sender immediately and destroy it. Any unauthorized copying, 
disclosure or distribution of the material in this e-mail is strictly 
forbidden. Tradeweb reserves the right to monitor all e-mail communications 
through its networks. If you do not wish to receive marketing emails about our 
products / services, please let us know by contacting us, either by email at 
contac...@tradeweb.com or by writing to us at the registered office of Tradeweb 
in the UK, which is: Tradeweb Europe Limited (company number 3912826), 1 Fore 
Street Avenue London EC2Y 9DT. To see our privacy policy, visit our website @ 
www.tradeweb.com.

Re: READ Queries timing out.

[Pranay akula]

Thanks ZAIDI,

Using C++ driver doesn't have tracing with driver so executing those from
cqlsh. when i am tracing i am getting below error, i increased
--request-timeout to 3600 in cqlsh.


> ReadTimeout: code=1200 [Coordinator node timed out waiting for replica
> nodes' responses] message="Operation timed out - received only 0
> responses." info={'received_responses': 0, 'required_responses': 1,
> 'consistency': 'ONE'}
> Statement trace did not complete within 10 seconds


The below are cfstats and cfhistograms, i can see  read latency, cell count
and Maximum live cells per slice (last five minutes) are high. is there any
way to get around this with out changing data model.

Percentile  SSTables Write Latency  Read Latency  Partition
> SizeCell Count
>  (micros)  (micros)
>(bytes)
> 50% 1.00 20.00   NaN
> 133120
> 75% 2.00 29.00   NaN
>686686
> 95% 8.00 60.00   NaN
>  126934  1331
> 98%10.00103.00   NaN
> 315852  3973
> 99%12.00149.00   NaN
>545791  8239
> Min 0.00  0.000.00
>   104 0
> Max20.00   12730764.00  9773372036884776000.00
> 74975550 83457




Read Count: 44514407
> Read Latency: 82.92876612928933 ms.
> Write Count: 3007585812
> Write Latency: 0.07094456590853208 ms.
> Pending Flushes: 0
> SSTable count: 9
> Space used (live): 66946214374
> Space used (total): 66946214374
> Space used by snapshots (total): 0
> Off heap memory used (total): 33706492
> SSTable Compression Ratio: 0.5598380206656697
> Number of keys (estimate): 2483819
> Memtable cell count: 15008
> Memtable data size: 330597
> Memtable off heap memory used: 518502
> Memtable switch count: 39915
> Local read count: 44514407
> Local read latency: 82.929 ms
> Local write count: 3007585849
> Local write latency: 0.071 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 12623632
> Bloom filter off heap memory used: 12623560
> Index summary off heap memory used: 3285614
> Compression metadata off heap memory used: 17278816
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 74975550
> Compacted partition mean bytes: 27111
> Average live cells per slice (last five minutes): 388.7486606077893
> Maximum live cells per slice (last five minutes): 28983.0
> Average tombstones per slice (last five minutes): 0.0
> Maximum tombstones per slice (last five minutes): 0.0



Thanks
Pranay.

On Fri, Jul 7, 2017 at 11:16 AM, Thakrar, Jayesh <
jthak...@conversantmedia.com> wrote:

> Can you provide more details.
>
> E.g. table structure, the app used for the query, the query itself and the
> error message.
>
>
>
> Also get the output of the following commands from your cluster nodes
> (note that one command uses "." and the other "space" between keyspace and
> tablename)
>
>
>
> nodetool -h  tablestats .
>
> nodetool -h  tablehistograms  
>
>
>
> Timeouts can happen at the client/application level (which can be tuned)
> and at the coordinator node level (which too can be tuned).
>
> But again those timeouts are a symptom of something.
>
> It can happen at the client side because of connection pool queue too full
> (which is likely due to response time from the cluster/coordinate nodes).
>
> And the issues at the cluster side could be due to several reasons.
>
> E.g. your query has to scan through too many tombstones, causing the delay
> or your query (if using filter).
>
>
>
> *From: *"ZAIDI, ASAD A" 
> *Date: *Friday, July 7, 2017 at 9:45 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *RE: READ Queries timing out.
>
>
>
> >> I analysed the GC logs not having any issues with major GC's
>
> If you don’t have issues on GC , than why do you want to
> [tune] GC parameters ?
>
> Instead focus on why select queries are taking time.. may be take a look
> on their trace?
>
>
>
>
>
> *From:* Pranay akula [mailto:pranay.akula2...@gmail.com]
> *Sent:* Friday, July 07, 2017 9:27 AM
> *To:* user@cassandra.apache.org
> *Subject:* READ Queries timing out.
>
>
>
> Lately i am seeing some select queries timing out, data modelling to blame
> for but not in a situation to redo it.
>
>
>
> Does increasing heap will help ??
>
>
>
> currently using 1GB new_heap, I analysed the GC logs not having any issues
> with major GC's .
>
>
>
> Using G1GC , does increasing new_heap will help ??
>
>
>
> currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i
> increase heap to lets say 2GB is that effective b/c young GC's will kick in
> m

"nodetool repair -dc"

[vasu gunja]

Hi ,

I have a question regarding "nodetool repair -dc" option. recently we added
multiple nodes to one DC center, we want to perform repair only on current
DC.

Here is my question.

Do we need to perform "nodetool repair -dc" on all nodes belongs to that DC
?
or only one node of that DC?



thanks,
V

Re: READ Queries timing out.

[Thakrar, Jayesh]

Can you provide more details.
E.g. table structure, the app used for the query, the query itself and the 
error message.

Also get the output of the following commands from your cluster nodes (note 
that one command uses "." and the other "space" between keyspace and tablename)

nodetool -h  tablestats .
nodetool -h  tablehistograms  

Timeouts can happen at the client/application level (which can be tuned) and at 
the coordinator node level (which too can be tuned).
But again those timeouts are a symptom of something.
It can happen at the client side because of connection pool queue too full 
(which is likely due to response time from the cluster/coordinate nodes).
And the issues at the cluster side could be due to several reasons.
E.g. your query has to scan through too many tombstones, causing the delay or 
your query (if using filter).

From: "ZAIDI, ASAD A" 
Date: Friday, July 7, 2017 at 9:45 AM
To: "user@cassandra.apache.org" 
Subject: RE: READ Queries timing out.

>> I analysed the GC logs not having any issues with major GC's
If you don’t have issues on GC , than why do you want to [tune] GC 
parameters ?
Instead focus on why select queries are taking time.. may be take a look on 
their trace?


From: Pranay akula [mailto:pranay.akula2...@gmail.com]
Sent: Friday, July 07, 2017 9:27 AM
To: user@cassandra.apache.org
Subject: READ Queries timing out.

Lately i am seeing some select queries timing out, data modelling to blame for 
but not in a situation to redo it.

Does increasing heap will help ??

currently using 1GB new_heap, I analysed the GC logs not having any issues with 
major GC's .

Using G1GC , does increasing new_heap will help ??

currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i 
increase heap to lets say 2GB is that effective b/c young GC's will kick in 
more frequently  to complete in 500ms right ??


Thanks
Pranay.

Re: Move Production data to Development Cluster

[Pranay akula]

Hello *Jonathan,*

As both clusters size is same.

 Do I copy the snapshots from all the nodes?  yes this will work, just make
sure that ur copying data to nodes with assiociated tokens.


Thanks
Pranay.

On Fri, Jul 7, 2017 at 10:48 AM, Jonathan Baynes <
jonathan.bay...@tradeweb.com> wrote:

> Hi,
>
>
>
> Can anyone help me. I’m trying (and failing) to move my 3 node C* data
> from my Production Environment to my Development 3 node cluster.
>
>
>
> Here is the fine print…
>
>
>
> Oracle Linux 7.3
>
> C* 3.0.11
>
>
>
> 3 Nodes ((virtual Nodes 256))
>
> 1 Keyspace (replication factor 3) Quorum Consistency
>
> 1 table
>
>
>
> Snapshot taken on each node.
>
>
>
> *Attempt 1*
>
>
>
> I’ve tried the following (http://docs.datastax.com/en/
> cassandra/3.0/cassandra/operations/opsSnapshotRestoreNewCluster.html)
>
>
>
>
>
>1. From the old cluster, retrieve the list of tokens associated with
>each node's IP:
>
> $ nodetool ring | grep ip_address_of_node | awk '{print $NF ","}' | xargs
>
>
>
> I’ve done this for all 3 nodes, placed them together in one string
>
>
>1. In the cassandra.yaml
>
> 
>  file
>for each node in the new cluster, add the list of tokens you obtained in
>the previous step to the initial_token
>
> 
>  parameter
>using the same *num_tokens* setting as in the old cluster.
>
> Added all the tokens from step one
>
>1. Make any other necessary changes in the new cluster's cassandra.yaml
>
> 
>  and
>property files so that the new nodes match the old cluster settings. Make
>sure the seed nodes are set for the new cluster.
>2. Clear the system table data from each new node:
>
> $ sudo rm -rf /var/lib/cassandra/data/system/*
>
> This allows the new nodes to use the initial tokens defined in the
> cassandra.yaml
> 
>  when
> they restart.
>
>1. Start each node using the specified list of token ranges in new
>cluster's cassandra.yaml
>
> 
>:
>
> initial_token: -9211270970129494930, -9138351317258731895, 
> -8980763462514965928, ...
>
>
>1. Create schema in the new cluster. All the schemas from the old
>cluster must be reproduced in the new cluster.
>2. Stop the node
>
> .
>Using nodetool refresh is unsafe because files within the data
>directory of a running node can be silently overwritten by identically
>named just-flushed SSTables from memtable flushes or compaction. Copying
>files into the data directory and restarting the node will not work for the
>same reason.
>3. Restore the SSTable files snapshotted
>
> 
>  from
>the old cluster onto the new cluster using the same directories, while
>noting that the UUID component of target directory names has changed.
>Without restoration, the new cluster will not have data to read upon
>restart.
>4. Restart the node.
>
>
>
> When  I restart I get errors in the Yaml file pointing to the token ranges.
>
>
>
> If I take out the tokens for 2 of the nodes and just use one node(s)
> tokens I can restore the data, but I get only a third of the row count id
> expect.
>
>
>
> I then noticed I was only restoring  the snapshot from that one node, so ,
> that made sense….
>
>
>
> So then I took all of the snapshots, from all of the nodes, placed them
> into a folder, re added all the tokens and re ran the process, but I get
> the token range error in the yaml again.
>
>
>
>
>
> *Attempt 2*
>
>
>
> So then I tried  SSTableLoader from the same folder with all the (3 nodes
> snapshots) and then I get corruption on the SSTable..
>
>
>
>
>
> *Advice*
>
>
>
> I’ve tried this so many ways its getting confusing.. what’s right?
>
>
>
> Can anyone give me some pointers as to the best route to migrate data from
> cluster to cluster? The Documentation is so vague and not detailed enough?
>
>
>
> Do I copy the snapshots from all the nodes?
>
> Do I just work on one node at a time?
>
>
>
> Any suggestions please??
>
>
>
> Thanks
>
> J
>
>
>
>
>
>
>
> *Jonathan Baynes*
>
> DBA
> Tradeweb Europe Limited
>
> Moor Place  •  1 Fore Street Avenue  •  London EC2Y 9DT
> P +44 (0)20 77760988 <+44%2020%

Re: READ Queries timing out.

[Pranay akula]

Thanks ZAIDI,

The problem is the tracing queries are also getting timed out, so not sure
how to troubleshoot.

Does increasing new_heap will help reads ?? what other param's i can tune,
so that i can identify the issue.


Thanks
Pranay.



On Fri, Jul 7, 2017 at 10:45 AM, ZAIDI, ASAD A  wrote:

> >> I analysed the GC logs not having any issues with major GC's
>
> If you don’t have issues on GC , than why do you want to
> [tune] GC parameters ?
>
> Instead focus on why select queries are taking time.. may be take a look
> on their trace?
>
>
>
>
>
> *From:* Pranay akula [mailto:pranay.akula2...@gmail.com]
> *Sent:* Friday, July 07, 2017 9:27 AM
> *To:* user@cassandra.apache.org
> *Subject:* READ Queries timing out.
>
>
>
> Lately i am seeing some select queries timing out, data modelling to blame
> for but not in a situation to redo it.
>
>
>
> Does increasing heap will help ??
>
>
>
> currently using 1GB new_heap, I analysed the GC logs not having any issues
> with major GC's .
>
>
>
> Using G1GC , does increasing new_heap will help ??
>
>
>
> currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i
> increase heap to lets say 2GB is that effective b/c young GC's will kick in
> more frequently  to complete in 500ms right ??
>
>
>
>
>
> Thanks
>
> Pranay.
>

Move Production data to Development Cluster

[Jonathan Baynes]

Hi,

Can anyone help me. I'm trying (and failing) to move my 3 node C* data from my 
Production Environment to my Development 3 node cluster.

Here is the fine print...

Oracle Linux 7.3
C* 3.0.11

3 Nodes ((virtual Nodes 256))
1 Keyspace (replication factor 3) Quorum Consistency
1 table

Snapshot taken on each node.

Attempt 1

I've tried the following 
(http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSnapshotRestoreNewCluster.html)



  1.  From the old cluster, retrieve the list of tokens associated with each 
node's IP:

$ nodetool ring | grep ip_address_of_node | awk '{print $NF ","}' | xargs



I've done this for all 3 nodes, placed them together in one string

  1.  In the 
cassandra.yaml
 file for each node in the new cluster, add the list of tokens you obtained in 
the previous step to the 
initial_token
 parameter using the same num_tokens setting as in the old cluster.
Added all the tokens from step one

  1.  Make any other necessary changes in the new cluster's 
cassandra.yaml
 and property files so that the new nodes match the old cluster settings. Make 
sure the seed nodes are set for the new cluster.
  2.  Clear the system table data from each new node:

$ sudo rm -rf /var/lib/cassandra/data/system/*

This allows the new nodes to use the initial tokens defined in the 
cassandra.yaml
 when they restart.

  1.  Start each node using the specified list of token ranges in new cluster's 
cassandra.yaml:

initial_token: -9211270970129494930, -9138351317258731895, 
-8980763462514965928, ...

  1.  Create schema in the new cluster. All the schemas from the old cluster 
must be reproduced in the new cluster.
  2.  Stop the 
node.
 Using nodetool refresh is unsafe because files within the data directory of a 
running node can be silently overwritten by identically named just-flushed 
SSTables from memtable flushes or compaction. Copying files into the data 
directory and restarting the node will not work for the same reason.
  3.  Restore the SSTable files 
snapshotted
 from the old cluster onto the new cluster using the same directories, while 
noting that the UUID component of target directory names has changed. Without 
restoration, the new cluster will not have data to read upon restart.
  4.  Restart the node.

When  I restart I get errors in the Yaml file pointing to the token ranges.

If I take out the tokens for 2 of the nodes and just use one node(s) tokens I 
can restore the data, but I get only a third of the row count id expect.

I then noticed I was only restoring  the snapshot from that one node, so , that 
made sense

So then I took all of the snapshots, from all of the nodes, placed them into a 
folder, re added all the tokens and re ran the process, but I get the token 
range error in the yaml again.


Attempt 2

So then I tried  SSTableLoader from the same folder with all the (3 nodes 
snapshots) and then I get corruption on the SSTable..


Advice

I've tried this so many ways its getting confusing.. what's right?

Can anyone give me some pointers as to the best route to migrate data from 
cluster to cluster? The Documentation is so vague and not detailed enough?

Do I copy the snapshots from all the nodes?
Do I just work on one node at a time?

Any suggestions please??

Thanks
J



Jonathan Baynes
DBA
Tradeweb Europe Limited
Moor Place  *  1 Fore Street Avenue  *  London EC2Y 9DT
P +44 (0)20 77760988  *  F +44 (0)20 7776 3201  *  M +44 (0) xx
jonathan.bay...@tradeweb.com

[cid:image001.jpg@01CD26AD.4165F110]   follow us:  
[cid:image002.jpg@01CD26AD.4165F110] 

[cid:image003.jpg@01CD26AD.4165F110] 
-
A leading marketplace for electronic 
fixed income, derivatives and ETF trading




This e-mail may contain confidential and/or privileged information. If you are 
not the intended recipient (or have received this e-mai

RE: READ Queries timing out.

[ZAIDI, ASAD A]

>> I analysed the GC logs not having any issues with major GC's
If you don’t have issues on GC , than why do you want to [tune] GC 
parameters ?
Instead focus on why select queries are taking time.. may be take a look on 
their trace?


From: Pranay akula [mailto:pranay.akula2...@gmail.com]
Sent: Friday, July 07, 2017 9:27 AM
To: user@cassandra.apache.org
Subject: READ Queries timing out.

Lately i am seeing some select queries timing out, data modelling to blame for 
but not in a situation to redo it.

Does increasing heap will help ??

currently using 1GB new_heap, I analysed the GC logs not having any issues with 
major GC's .

Using G1GC , does increasing new_heap will help ??

currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i 
increase heap to lets say 2GB is that effective b/c young GC's will kick in 
more frequently  to complete in 500ms right ??


Thanks
Pranay.

RE: Understanding of cassandra metrics

[ZAIDI, ASAD A]

What exactly does mean CoordinatorScanLatency for example
CoordinatorScanLatency  is a timer metric that present coordinator range scan 
latency for  table.
Is it latency on full table scan or maybe range scan by clustering key?
It is range scan.. clustering key is used to only store data in 
sorted fashion – partition key along with chosen partitioner helps in range 
scan of data.
Can anybody write into partition while locked?
Writes are atomic – it depends on your chosen consistency level 
to determine if writes will fail or succeed.

From: Павел Сапежко [mailto:amelius0...@gmail.com]
Sent: Friday, July 07, 2017 8:23 AM
To: user@cassandra.apache.org
Subject: Re: Understanding of cassandra metrics

Are you really think that I don't read docs? Do you have enough information in 
the documentation? I think no. What exactly does mean CoordinatorScanLatency 
for example? Is it latency on full table scan or maybe range scan by clustering 
key? What exactly mean ViewLockAcquireTime? What is "partition lock"? Can 
anybody write into partition while locked? Etc.
пт, 7 июл. 2017 г. в 13:01, Ivan Iliev 
mailto:ivan.iliev.il...@gmail.com>>:
1st result on google returns:

http://cassandra.apache.org/doc/latest/operating/metrics.html

On Fri, Jul 7, 2017 at 12:16 PM, Павел Сапежко 
mailto:amelius0...@gmail.com>> wrote:
Hello, I have several question about cassandra metrics. What does exactly mean 
the next metrics:

  *   CoordinatorReadLatency
  *   CoordinatorScanLatency
  *   ReadLatency
  *   RangeLatency
  *   ViewLockAcquireTime
  *   ViewReadTime
--

С уважением,

Павел Сапежко

skype: p.sapezhko

--

С уважением,

Павел Сапежко

skype: p.sapezhko

READ Queries timing out.

[Pranay akula]

Lately i am seeing some select queries timing out, data modelling to blame
for but not in a situation to redo it.

Does increasing heap will help ??

currently using 1GB new_heap, I analysed the GC logs not having any issues
with major GC's .

Using G1GC , does increasing new_heap will help ??

currently using JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500", even if i
increase heap to lets say 2GB is that effective b/c young GC's will kick in
more frequently  to complete in 500ms right ??


Thanks
Pranay.

Re: Understanding of cassandra metrics

[Павел Сапежко]

Are you really think that I don't read docs? Do you have enough information
in the documentation? I think no. What exactly does mean CoordinatorScanLatency
for example? Is it latency on full table scan or maybe range scan by
clustering key? What exactly mean ViewLockAcquireTime? What is "partition
lock"? Can anybody write into partition while locked? Etc.
пт, 7 июл. 2017 г. в 13:01, Ivan Iliev :

> 1st result on google returns:
>
> http://cassandra.apache.org/doc/latest/operating/metrics.html
>
> On Fri, Jul 7, 2017 at 12:16 PM, Павел Сапежко 
> wrote:
>
>> Hello, I have several question about cassandra metrics. What does exactly
>> mean the next metrics:
>>
>>- CoordinatorReadLatency
>>- CoordinatorScanLatency
>>- ReadLatency
>>- RangeLatency
>>- ViewLockAcquireTime
>>- ViewReadTime
>>
>> --
>>
>> С уважением,
>>
>> Павел Сапежко
>>
>> skype: p.sapezhko
>>
>
> --

С уважением,

Павел Сапежко

skype: p.sapezhko

Re: Cannot achieve consistency level LOCAL_ONE

[Oleksandr Shulgin]

On Thu, Jul 6, 2017 at 6:58 PM, Charulata Sharma (charshar) <
chars...@cisco.com> wrote:

> Hi,
>
> I am facing similar issues with SYSTEM_AUTH keyspace and wanted to know
> the implication of disabling the "*cassandra*" superuser.
>

Unless you have scheduled any tasks that require the user with that name to
be there, there are no implications.  This user is not used by Cassandra
tools or the server process internally, so nothing really depends on it.

Of course, in order to drop a superuser account, you need to create another
superuser, so in the end you still have superuser access to your cluster.

Cheers,
--
Alex

Re: Understanding of cassandra metrics

[Ivan Iliev]

1st result on google returns:

http://cassandra.apache.org/doc/latest/operating/metrics.html

On Fri, Jul 7, 2017 at 12:16 PM, Павел Сапежко 
wrote:

> Hello, I have several question about cassandra metrics. What does exactly
> mean the next metrics:
>
>- CoordinatorReadLatency
>- CoordinatorScanLatency
>- ReadLatency
>- RangeLatency
>- ViewLockAcquireTime
>- ViewReadTime
>
> --
>
> С уважением,
>
> Павел Сапежко
>
> skype: p.sapezhko
>

Understanding of cassandra metrics

[Павел Сапежко]

Hello, I have several question about cassandra metrics. What does exactly
mean the next metrics:

   - CoordinatorReadLatency
   - CoordinatorScanLatency
   - ReadLatency
   - RangeLatency
   - ViewLockAcquireTime
   - ViewReadTime

-- 

С уважением,

Павел Сапежко

skype: p.sapezhko

Re: Corrupted commit log prevents Cassandra start

[Hannu Kröger]

Hello,

yes, that’s what we do when things like this happen.

My thinking is just that when commit log is corrupted, you cannot really do
anything else but exactly those steps. Delete corrupted file and run repair
after starting. At least I haven’t heard of any tools for salvaging commit
log sections.

Current behaviour gives DBA control over when to do those things and of
course DBA realizes this way that things didn’t go ok but that’s about it.
There is no alternative way of healing the system or anything.

Hannu

On 7 July 2017 at 12:03:06, benjamin roth (brs...@gmail.com) wrote:

Hi Hannu,

I remember there have been discussions about this in the past. Most
probably there is already a JIRA for this.
I roughly remember a consense like that:
- Default behaviour should remain
- It should be configurable to the needs and preferences of the DBA
- It should at least spit out errors in the logs

... of course it would be even better to have the underlying issue fixed
that commit logs should not be corrupt but I remember that this is not so
easy due to some "architectural implications" of Cassandra. IIRC Ed
Capriolo posted something related to that some months ago.

For a quick fix, I'd recommend:
- Delete the affected log file
- Start the node
- Run a full-range (not -pr) repair on that node

2017-07-07 10:57 GMT+02:00 Hannu Kröger :

> Hello,
>
> We had a test server crashing for some reason (not related to Cassandra
> probably) and now when trying to start cassandra, it gives following error:
>
> ERROR [main] 2017-07-06 09:29:56,140 JVMStabilityInspector.java:82 -
> Exiting due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Mutation checksum failure at 24240116 in Next section at 24239690 in
> CommitLog-6-1498576271195.log
> at org.apache.cassandra.db.commitlog.CommitLogReader.
> readSection(CommitLogReader.java:332) [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:201)
> [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.db.commitlog.CommitLogReader.
> readAllFiles(CommitLogReader.java:84) [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.db.commitlog.CommitLogReplayer.
> replayFiles(CommitLogReplayer.java:140) [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.db.commitlog.CommitLog.
> recoverFiles(CommitLog.java:177) [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:158)
> [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:326)
> [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
> [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735)
> [apache-cassandra-3.10.jar:3.10]
>
> Shouldn’t Cassandra tolerate this situation?
>
> Of course we can delete commit logs and life goes on. But isn’t this a bug
> or something?
>
> Hannu
>
>

Re: Corrupted commit log prevents Cassandra start

[benjamin roth]

Hi Hannu,

I remember there have been discussions about this in the past. Most
probably there is already a JIRA for this.
I roughly remember a consense like that:
- Default behaviour should remain
- It should be configurable to the needs and preferences of the DBA
- It should at least spit out errors in the logs

... of course it would be even better to have the underlying issue fixed
that commit logs should not be corrupt but I remember that this is not so
easy due to some "architectural implications" of Cassandra. IIRC Ed
Capriolo posted something related to that some months ago.

For a quick fix, I'd recommend:
- Delete the affected log file
- Start the node
- Run a full-range (not -pr) repair on that node

2017-07-07 10:57 GMT+02:00 Hannu Kröger :

> Hello,
>
> We had a test server crashing for some reason (not related to Cassandra
> probably) and now when trying to start cassandra, it gives following error:
>
> ERROR [main] 2017-07-06 09:29:56,140 JVMStabilityInspector.java:82 -
> Exiting due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Mutation checksum failure at 24240116 in Next section at 24239690 in
> CommitLog-6-1498576271195.log
> at org.apache.cassandra.db.commitlog.CommitLogReader.
> readSection(CommitLogReader.java:332) [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:201)
> [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.db.commitlog.CommitLogReader.
> readAllFiles(CommitLogReader.java:84) [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.db.commitlog.CommitLogReplayer.
> replayFiles(CommitLogReplayer.java:140) [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.db.commitlog.CommitLog.
> recoverFiles(CommitLog.java:177) [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:158)
> [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:326)
> [apache-cassandra-3.10.jar:3.10]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
> [apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735)
> [apache-cassandra-3.10.jar:3.10]
>
> Shouldn’t Cassandra tolerate this situation?
>
> Of course we can delete commit logs and life goes on. But isn’t this a bug
> or something?
>
> Hannu
>
>

Corrupted commit log prevents Cassandra start

[Hannu Kröger]

Hello,

We had a test server crashing for some reason (not related to Cassandra
probably) and now when trying to start cassandra, it gives following error:

ERROR [main] 2017-07-06 09:29:56,140 JVMStabilityInspector.java:82 -
Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
Mutation checksum failure at 24240116 in Next section at 24239690 in
CommitLog-6-1498576271195.log
at
org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:332)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:201)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:84)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:140)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:177)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:158)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:326)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
[apache-cassandra-3.10.jar:3.10]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735)
[apache-cassandra-3.10.jar:3.10]

Shouldn’t Cassandra tolerate this situation?

Of course we can delete commit logs and life goes on. But isn’t this a bug
or something?

Hannu

resultSetMetadata returned by PreparedStatement for sum(int) should be bigint

[Daniel Woo]

Hi guys,

I am trying to calculate sum(int) with a prepared statement. The problem is
that I have many rows to sum up, the result could be a bigint, however
cassandra java driver receives the metadata as int. My current workaround
is to change the column type to bigint.

What's the best practice for this?

-- 
Thanks & Regards,
Daniel