RE: cassandra reads are unbalanced

[Walsh, Stephen]

What client Cassandra driver are you using? Java?
Java driver 2.1.8

Is there only a single thread in each client or are there multiple threads
Multi in parallel.

What does your connection code look like
It’s a very large class based on config files, but I believe you’re interested 
in this line


cluster.withLoadBalancingPolicy(
  new 
DCAwareRoundRobinPolicy(config.getString(ConfigurationKeys.CassandraDataCenterName),

config.getInt(ConfigurationKeys.CassandraFailoverDataCenterNodesToLookAt),true))

with each of our application having a different (local) Datacenter name.

Steve

From: Jack Krupansky [mailto:jack.krupan...@gmail.com]
Sent: 04 December 2015 16:46
To: user@cassandra.apache.org
Subject: Re: cassandra reads are unbalanced

Thanks for the elaboration. A few more questions...

Is there only a single thread in each client or are there multiple threads 
doing reading in parallel? IOW, does a read need to complete before the next 
read is issued.

What client Cassandra driver are you using? Java?

What does your connection code look like, say compared to the example in the 
doc:
http://docs.datastax.com/en/developer/java-driver/2.0/java-driver/quick_start/qsSimpleClientCreate_t.html

Just to make sure it really is connecting only to the local cluster and using 
round robin and whether it is token aware.


-- Jack Krupansky

On Fri, Dec 4, 2015 at 10:51 AM, Walsh, Stephen 
mailto:stephen.wa...@aspect.com>> wrote:
Thanks for your input, but I think I’ve already answered most of your questions.


How many clients do you have performing reads?

--
On Wed, Dec 2, 2015 at 6:44 PM, Walsh, Stephen 
mailto:stephen.wa...@aspect.com>> wrote
….
There are 2 application (1 for each DC) who read and write at the same rate to 
their local DC
….







Is your load balancer in front of your clients or between your clients and 
Cassandra?

--
On Thu, Dec 3, 2015 at 4:58 AM, Walsh, Stephen 
mailto:stephen.wa...@aspect.com>> wrote:
…
our production applications are behind a round robin load balancer
…
--

No Load Balancers talk to cassandra – I’m only mentioning this to show that the 
writes / read are evenly distributed over the 2 DC’s






Does Node1 of DC2 have the exact same configuration of hardware of the other 
nodes
Yes





Is it in the same rack
It’s in AWS – but we have it configured via the GossipProperytFileSnitch that 
they are all on unique racks






Maybe your load balancer thinks that node is more capable and handles requests 
faster so that it looks less loaded than the other two nodes
Unlikely, it’s all TCP SSL pass though connections. It doesn’t balance on load, 
it just round robins each request





You might also check the read counts after a very short interval of time to see 
if Node1 is uniformly getting more requests or just occasionally
--
On Wed, Dec 2, 2015 at 3:36 PM, Walsh, Stephen 
mailto:stephen.wa...@aspect.com>> wrote …
We monitor the number of reads / writes of every table via the cassandra JMX 
metrics. (cassandra.db.read_count)
…
--
We can only monitor in 1 hour moving window




Maybe the other two nodes are in a different rack that occasionally has net 
connectivity issues
Unlikely seems its AWS






From: Jack Krupansky 
[mailto:jack.krupan...@gmail.com]
Sent: 03 December 2015 16:11

To: user@cassandra.apache.org
Subject: Re: cassandra reads are unbalanced

How many clients do you have performing reads?

Is your load balancer in front of your clients or between your clients and 
Cassandra?

Does Node1 of DC2 have the exact same configuration of hardware of the other 
nodes? Is it in the same rack? Maybe your load balancer thinks that node is 
more capable and handles requests faster so that it looks less loaded than the 
other two nodes.

You might also check the read counts after a very short interval of time to see 
if Node1 is uniformly getting more requests or just occasionally. Maybe the 
other two nodes are in a different rack that occasionally has net connectivity 
issues so that the requests get diverted by the client/load balancer to Node1 
during those times.


-- Jack Krupansky

On Thu, Dec 3, 2015 at 4:58 AM, Walsh, Stephen 
mailto:stephen.wa...@aspect.com>> wrote:
Thanks but keep in mind that both DC should be getting the same load, our 
production applications are behind a round robin load balancer – so each one 
our local application talk to its local Cassandra DataCenter.

It took about 4 hours but the nodetool cleanup eventually balanced all nodes

From: DuyHai Doan [mailto:doanduy...@gmail.com]
Sent: 02 December 2015 16:27

To: user@cassandra.apache.org
Subject: Re: cassandra reads are unbalanced

If you're using the Java driver with LOCAL_ONE and the default load balancing 
strategy (TokenAware wrapped on DCAwareRoundRobin

Re: AttributeError python exception when using Models

[Adam Holmberg]

This is a good question for the Python driver mailing list
.
(changing recipients on this message)

To answer your question: this is happening because there is no connection
setup for the cqlengine mapper. When using cqlengine, you must setup the
connection before querying or using management functions:
http://datastax.github.io/python-driver/object_mapper.html#getting-started
http://datastax.github.io/python-driver/api/cassandra/cqlengine/connection.html

That said, this condition could be handled better. I've created a ticket to
improve the error message in these circumstances:
https://datastax-oss.atlassian.net/browse/PYTHON-451

Regards,
Adam Holmberg

Re: Handle Node Failure with Repair -pr

[Anuj Wadehra]

Hi All !!!


Any comments on the repair -pr scenarios..please share how you deal with such 
scenarios..


Thanks

Anuj

Sent from Yahoo Mail on Android

From:"Anuj Wadehra" 
Date:Sat, 5 Dec, 2015 at 12:57 am
Subject:Handle Node Failure with Repair -pr

Hi Guys !!


I need comments on my understanding of repair -pr ..If you are using repair -pr 
in your cluster then following statements hold true:


1. If a node goes down for long time and your not sure when will it return, you 
must ensure that subrange repair for the defected node range is done within 
gc_grace_period from some other node?


 I think the mandatory requirement for repair must be restated to make it 
explicit. While saying that each node must run repair -pr within gc grace, we 
must clearly mention that each node' s range must be repaired and care must be 
taken to run subrange repair from separate node in case a node is down and gc 
grace is approaching.Otherwise no repair -pr job on nodes will repair that 
subrange even though all live nodes were meeting the norm of running repair -pr 
within gc grace.


2. If you forgot to run repair -pr within gc grace seconds on one of the nodes, 
deleting data folder and autobootstrapping will not help as subrange for node 
was never repaired and any node with missed delete will popup the data back.You 
can only minimize deletes from popping up but cant prevent them completely.


Thanks

Anuj



Sent from Yahoo Mail on Android

[RELEASE] Apache Cassandra 2.2.4 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 2.2.4.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 2.2 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: http://goo.gl/EWjhm1 (CHANGES.txt)
[2]: http://goo.gl/WLSytN (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

[RELEASE] Apache Cassandra 2.1.12 released

[Jake Luciani]

The Cassandra team is pleased to announce the release of Apache Cassandra
version 2.1.12.

Apache Cassandra is a fully distributed database. It is the right choice
when you need scalability and high availability without compromising
performance.

 http://cassandra.apache.org/

Downloads of source and binary distributions are listed in our download
section:

 http://cassandra.apache.org/download/

This version is a bug fix release[1] on the 2.1 series. As always, please
pay
attention to the release notes[2] and Let us know[3] if you were to
encounter
any problem.

Enjoy!

[1]: http://goo.gl/Phl5Pd (CHANGES.txt)
[2]: http://goo.gl/L1HIfj (NEWS.txt)
[3]: https://issues.apache.org/jira/browse/CASSANDRA

lots of tombstone after compaction

[Kai Wang]

I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data are
large amount of wide rows with lots of null's. It takes one day or two for
the compaction to complete. sstable count is at single digit. Maximum
partition size is ~50M and mean size is ~5M. However I am seeing frequent
read query timeouts caused by tombstone_failure_threshold (10). These
tables are basically read-only. There're no writes.

I just kicked off compaction on those tables using nodetool. Hopefully it
can remove those tombstones. But is it normal to have these many tombstones
after the initial compactions? Is this related to the fact the original
data has lots of nulls?

Thanks.

Re: lots of tombstone after compaction

[Robert Wille]

The nulls in the original data created the tombstones. They won’t go away until 
gc_grace_seconds have passed (default is 10 days).

On Dec 7, 2015, at 4:46 PM, Kai Wang  wrote:

> I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data are 
> large amount of wide rows with lots of null's. It takes one day or two for 
> the compaction to complete. sstable count is at single digit. Maximum 
> partition size is ~50M and mean size is ~5M. However I am seeing frequent 
> read query timeouts caused by tombstone_failure_threshold (10). These 
> tables are basically read-only. There're no writes.
> 
> I just kicked off compaction on those tables using nodetool. Hopefully it can 
> remove those tombstones. But is it normal to have these many tombstones after 
> the initial compactions? Is this related to the fact the original data has 
> lots of nulls?
> 
> Thanks.

Re: lots of tombstone after compaction

[Jeff Jirsa]

https://issues.apache.org/jira/browse/CASSANDRA-7953

https://issues.apache.org/jira/browse/CASSANDRA-10505

There are buggy versions of cassandra that will multiple tombstones during 
compaction. 2.1.12 SHOULD correct that, if you’re on 2.1.



From:  Kai Wang
Reply-To:  "user@cassandra.apache.org"
Date:  Monday, December 7, 2015 at 3:46 PM
To:  "user@cassandra.apache.org"
Subject:  lots of tombstone after compaction

I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data are 
large amount of wide rows with lots of null's. It takes one day or two for the 
compaction to complete. sstable count is at single digit. Maximum partition 
size is ~50M and mean size is ~5M. However I am seeing frequent read query 
timeouts caused by tombstone_failure_threshold (10). These tables are 
basically read-only. There're no writes.

I just kicked off compaction on those tables using nodetool. Hopefully it can 
remove those tombstones. But is it normal to have these many tombstones after 
the initial compactions? Is this related to the fact the original data has lots 
of nulls?

Thanks.



smime.p7s
Description: S/MIME cryptographic signature

Re: Cassandra compaction stuck? Should I disable?

[Kai Wang]

Thank you for the investigation.
On Dec 2, 2015 5:21 AM, "PenguinWhispererThe ." <
th3penguinwhispe...@gmail.com> wrote:

> So it seems I found the problem.
>
> The node opening a stream is waiting for the other node to respond but
> that node never responds due to a broken pipe which makes Cassandra wait
> forever.
>
> It's basically this issue:
> https://issues.apache.org/jira/browse/CASSANDRA-8472
> And this is the workaround/fix:
> https://issues.apache.org/jira/browse/CASSANDRA-8611
>
> So:
> - update cassandra to >=2.0.11
> - add option streaming_socket_timeout_in_ms = 1
> - do rolling restart of cassandra
>
> What's weird is that the IOException: Broken pipe is never shown in my
> logs (not on any node). And my logging is set to INFO in log4j config.
> I have this config in log4j-server.properties:
> # output messages into a rolling log file as well as stdout
> log4j.rootLogger=INFO,stdout,R
>
> # stdout
> log4j.appender.stdout=org.apache.log4j.ConsoleAppender
> log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
> log4j.appender.stdout.layout.ConversionPattern=%5p %d{HH:mm:ss,SSS} %m%n
>
> # rolling log file
> log4j.appender.R=org.apache.log4j.RollingFileAppender
> log4j.appender.R.maxFileSize=20MB
> log4j.appender.R.maxBackupIndex=50
> log4j.appender.R.layout=org.apache.log4j.PatternLayout
> log4j.appender.R.layout.ConversionPattern=%5p [%t] %d{ISO8601} %F (line
> %L) %m%n
> # Edit the next line to point to your logs directory
> log4j.appender.R.File=/var/log/cassandra/system.log
>
> # Application logging options
> #log4j.logger.org.apache.cassandra=DEBUG
> #log4j.logger.org.apache.cassandra.db=DEBUG
> #log4j.logger.org.apache.cassandra.service.StorageProxy=DEBUG
>
> # Adding this to avoid thrift logging disconnect errors.
> log4j.logger.org.apache.thrift.server.TNonblockingServer=ERROR
>
> Too bad nobody else could point to those. Hope it helps someone else from
> wasting a lot of time.
>
> 2015-11-11 15:42 GMT+01:00 Sebastian Estevez <
> sebastian.este...@datastax.com>:
>
>> Use 'nodetool compactionhistory'
>>
>> all the best,
>>
>> Sebastián
>> On Nov 11, 2015 3:23 AM, "PenguinWhispererThe ." <
>> th3penguinwhispe...@gmail.com> wrote:
>>
>>> Does compactionstats shows only stats for completed compactions (100%)?
>>> It might be that the compaction is running constantly, over and over again.
>>> In that case I need to know what I might be able to do to stop this
>>> constant compaction so I can start a nodetool repair.
>>>
>>> Note that there is a lot of traffic on this columnfamily so I'm not sure
>>> if temporary disabling compaction is an option. The repair will probably
>>> take long as well.
>>>
>>> Sebastian and Rob: do you might have any more ideas about the things I
>>> put in this thread? Any help is appreciated!
>>>
>>> 2015-11-10 20:03 GMT+01:00 PenguinWhispererThe . <
>>> th3penguinwhispe...@gmail.com>:
>>>
 Hi Sebastian,

 Thanks for your response.

 No swap is used. No offense, I just don't see a reason why having swap
 would be the issue here. I put swapiness on 1. I also have jna installed.
 That should prevent java being swapped out as wel AFAIK.


 2015-11-10 19:50 GMT+01:00 Sebastian Estevez <
 sebastian.este...@datastax.com>:

> Turn off Swap.
>
>
> http://docs.datastax.com/en/cassandra/2.1/cassandra/install/installRecommendSettings.html?scroll=reference_ds_sxl_gf3_2k__disable-swap
>
>
> All the best,
>
>
> [image: datastax_logo.png] 
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com
>
> [image: linkedin.png]  [image:
> facebook.png]  [image: twitter.png]
>  [image: g+.png]
> 
> 
> 
>
>
> 
>
> DataStax is the fastest, most scalable distributed database
> technology, delivering Apache Cassandra to the world’s most innovative
> enterprises. Datastax is built to be agile, always-on, and predictably
> scalable to any size. With more than 500 customers in 45 countries, 
> DataStax
> is the database technology and transactional backbone of choice for the
> worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Tue, Nov 10, 2015 at 1:48 PM, PenguinWhispererThe . <
> th3penguinwhispe...@gmail.com> wrote:
>
>> I also have the following memory usage:
>> [root@US-BILLINGDSX4 cassandra]# free -m
>>  total   used   free sharedbuffers
>> cached
>> Mem: 12024   9455   2569  0110
>> 2163
>> -/+ buffers/cache:   7180   4844
>> Swap:

Re: lots of tombstone after compaction

[Kai Wang]

Rob and Jeff,

Thank you. It makes sense. I am on 2.1.10 and will upgrade to 2.1.12.
On Dec 7, 2015 7:05 PM, "Jeff Jirsa"  wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-7953
>
> https://issues.apache.org/jira/browse/CASSANDRA-10505
>
> There are buggy versions of cassandra that will multiple tombstones during
> compaction. 2.1.12 SHOULD correct that, if you’re on 2.1.
>
>
>
> From: Kai Wang
> Reply-To: "user@cassandra.apache.org"
> Date: Monday, December 7, 2015 at 3:46 PM
> To: "user@cassandra.apache.org"
> Subject: lots of tombstone after compaction
>
> I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data
> are large amount of wide rows with lots of null's. It takes one day or two
> for the compaction to complete. sstable count is at single digit. Maximum
> partition size is ~50M and mean size is ~5M. However I am seeing frequent
> read query timeouts caused by tombstone_failure_threshold (10). These
> tables are basically read-only. There're no writes.
>
> I just kicked off compaction on those tables using nodetool. Hopefully it
> can remove those tombstones. But is it normal to have these many tombstones
> after the initial compactions? Is this related to the fact the original
> data has lots of nulls?
>
> Thanks.
>

Re: AttributeError python exception when using Models

[Юрий Пухальский]

Thank you, Adam!

For some reason I missed that part, thought that the created session will 
automatically be used, silly me:)
But indeed, the error handling might have been better.

Adam Holmberg  writes:

> This is a good question for the Python driver mailing list. (changing
> recipients on this message)
>
> To answer your question: this is happening because there is no
> connection setup for the cqlengine mapper. When using cqlengine, you
> must setup the connection before querying or using management
> functions:
> http://datastax.github.io/python-driver/object_mapper.html#
> getting-started
> http://datastax.github.io/python-driver/api/cassandra/cqlengine/
> connection.html
>
> That said, this condition could be handled better. I've created a
> ticket to improve the error message in these circumstances:
> https://datastax-oss.atlassian.net/browse/PYTHON-451
>
> Regards,
> Adam Holmberg