Re: High GC activity on node with 4TB on data
Hi Jiri,
We do run multiple nodes with 2TB to 4TB of data and we will usually see GC
pressure when we create a lot of tombstones.
With Cassandra 2.0.x you would be able to see a log with the following pattern:
WARN [ReadStage:7] 2015-02-08 22:55:09,621 SliceQueryFilter.java (line 225)
Read 939 live and 1017 tombstoned cells in SyncCore.ContactInformation (see
tombstone_warn_threshold). 1000 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
This basically indicates that you add some major deletions for a given row.
Thanks,
FR
From: Mark Reddy mark.l.re...@gmail.com
To: user@cassandra.apache.org
Cc: cassandra-u...@apache.org; FF Systems ff-sys...@avast.com
Sent: Sunday, February 8, 2015 1:32 PM
Subject: Re: High GC activity on node with 4TB on data
Hey Jiri,
While I don't have any experience running 4TB nodes (yet), I would recommend
taking a look at a presentation by Arron Morton on large nodes:
http://planetcassandra.org/blog/cassandra-community-webinar-videoslides-large-nodes-with-cassandra-by-aaron-morton/
to see if you can glean anything from that.
I would note that at the start of his talk he mentions that in version 1.2 we
can now talk about nodes around 1 - 3 TB in size, so if you are storing
anything more than that you are getting into very specialised use cases.
If you could provide us with some more information about your cluster setup
(No. of CFs, read/write patterns, do you delete / update often, etc.) that may
help in getting you to a better place.
Regards,
Mark
On 8 February 2015 at 21:10, Kevin Burton bur...@spinn3r.com wrote:
Do you have a lot of individual tables? Or lots of small compactions?
I think the general consensus is that (at least for Cassandra), 8GB heaps are
ideal.
If you have lots of small tables it’s a known anti-pattern (I believe) because
the Cassandra internals could do a better job on handling the in memory
metadata representation.
I think this has been improved in 2.0 and 2.1 though so the fact that you’re on
1.2.18 could exasperate the issue. You might want to consider an upgrade
(though that has its own issues as well).
On Sun, Feb 8, 2015 at 12:44 PM, Jiri Horky ho...@avast.com wrote:
Hi all,
we are seeing quite high GC pressure (in old space by CMS GC Algorithm)
on a node with 4TB of data. It runs C* 1.2.18 with 12G of heap memory
(2G for new space). The node runs fine for couple of days when the GC
activity starts to raise and reaches about 15% of the C* activity which
causes dropped messages and other problems.
Taking a look at heap dump, there is about 8G used by SSTableReader
classes in org.apache.cassandra.io.compress.CompressedRandomAccessReader.
Is this something expected and we have just reached the limit of how
many data a single Cassandra instance can handle or it is possible to
tune it better?
Regards
Jiri Horky
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
blog: http://burtonator.wordpress.com… or check out my Google+ profile
Advice on how to handle corruption in system/hints
Hi everyone, We are running some Cassandra clusters (Usually a cluster of 5 nodes with replication factor of 3.) And at least once per day we do see some corruption related to a specific sstable in system/hints. (We are using Cassandra version 1.2.16 on RHEL 6.5) Here is an example of such exception: ERROR [CompactionExecutor:1694] 2014-06-08 21:37:33,267 CassandraDaemon.java (line 191) Exception in thread Thread[CompactionExecutor:1694,1,main] org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSize of 8224262783474088549 starting at 502360510 would be larger than file /home/y/var/cassandra/data/syste m/hints/system-hints-ic-281-Data.db length 504590769 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:167) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:83) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:69) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:180) at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142) at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145) at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:442) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: dataSize of 8224262783474088549 starting at 502360510 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-281-Data.db length 504590769 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:123) ... 23 more INFO [HintedHandoff:35] 2014-06-08 21:37:33,267 HintedHandOffManager.java (line 296) Started hinted handoff for host: 502a48cd-171b-4e83-a9ad-67f32437353a with IP: /10.210.239.190 ERROR [HintedHandoff:33] 2014-06-08 21:37:33,267 CassandraDaemon.java (line 191) Exception in thread Thread[HintedHandoff:33,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSize of 8224262783474088549 starting at 502360510 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-281-Data.db length 504590769 at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90) at org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSize of 8224262783474088549 starting at 502360510 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-281-Data.db length 504590769 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at
Re: Many to one type of replication.
Thanks much, I wanted to confirm. We will do this at the application level. FR On Sun, Mar 24, 2013 at 10:03 AM, aaron morton aa...@thelastpickle.comwrote: From this mailing list I found this Github project that is doing something similar by looking at the commit logs: https://github.com/carloscm/cassandra-commitlog-extract IMHO tailing the logs is fragile, and you may be better off handling it at the application level. But is there other options around using a custom replication strategy? There is no such thing as one directional replication. For example replication everything from DC 1 to DC 2, but do not replicate from DC 2 to DC 1. You may be better off reducing the number of clusters and then running one transactional and one analytical DC. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 24/03/2013, at 3:42 AM, Francois Richard frich...@xobni.com wrote: Hi, We currently run our Cassandra deployment with multiple independent clusters. The clusters are totally self contain in terms of redundancy and independent from each others. We have a sharding layer higher in our stack to dispatch the requests to the right application stack and this stack connects to his associated Cassandra cluster. All the cassandra clusters are identical in terms of hosted keyspaces, column families, replication factor ... At this point I am investigating ways to build a central cassandra cluster that could contain all the data from all the other cassandra clusters and I am wondering how to best do it. The goal is to have a global view of our data and to be able to do some massive crunching on it. For sure we can build some ETL type of job that would figure out the data that was updated, extract it, and load it to the central cassandra cluster. From this mailing list I found this Github project that is doing something similar by looking at the commit logs: https://github.com/carloscm/cassandra-commitlog-extract But is there other options around using a custom replication strategy? Any other general suggestions ? Thanks, FR -- _ *Francois Richard * -- _ *Francois Richard * VP Server Engineering and Operations** Xobni Engineering Xobni, Inc. 539 Bryant St San Francisco, CA 94107 415-987-5305 Mobile (For emergencies please leave a voice-mail to mobile) www.xobni.com**
Many to one type of replication.
Hi, We currently run our Cassandra deployment with multiple independent clusters. The clusters are totally self contain in terms of redundancy and independent from each others. We have a sharding layer higher in our stack to dispatch the requests to the right application stack and this stack connects to his associated Cassandra cluster. All the cassandra clusters are identical in terms of hosted keyspaces, column families, replication factor ... At this point I am investigating ways to build a central cassandra cluster that could contain all the data from all the other cassandra clusters and I am wondering how to best do it. The goal is to have a global view of our data and to be able to do some massive crunching on it. For sure we can build some ETL type of job that would figure out the data that was updated, extract it, and load it to the central cassandra cluster. From this mailing list I found this Github project that is doing something similar by looking at the commit logs: https://github.com/carloscm/cassandra-commitlog-extract But is there other options around using a custom replication strategy? Any other general suggestions ? Thanks, FR -- _ *Francois Richard *
RE: do I need to add more nodes? minor compaction eat all IO
Jonathan, Are you sure that the reads done for compaction are sequential with Cassandra 0.6.13? This is not what I am observing right now. During a minor compaction I usually observe ~ 1500 to 1900 r/s while rMB/s is barely around 30 to 35MB/s. Just asking out of curiosity. FR -Original Message- From: Jonathan Ellis [mailto:jbel...@gmail.com] Sent: Saturday, July 23, 2011 5:05 PM To: user@cassandra.apache.org Subject: Re: do I need to add more nodes? minor compaction eat all IO On Sat, Jul 23, 2011 at 4:16 PM, Francois Richard frich...@xobni.com wrote: My understanding is that during compaction cassandra does a lot of non sequential readsa then dumps the results with a big sequential write. Compaction reads and writes are both sequential, and 0.8 allows setting a MB/s to cap compaction at. As to the original question do I need to add more machines I'd say that depends more on whether your application's SLA is met, than what % io util spikes to. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
RE: do I need to add more nodes? minor compaction eat all IO
This really depends on your disks setup. When you run iostat under high load, do you see a high number of r/s but the rMB/s is not so great? I usually use: iostat -x -m sdb sdc 1 to monitor situation like this. In my case my disk setup is the following: OS -- /sda Cassandra CommitLogs -- /sdb Cassandra Data -- /sdc My understanding is that during compaction cassandra does a lot of non sequential readsa then dumps the results with a big sequential write. Is your application mostly doing writes and little reads or the other way around. FR -Original Message- From: Yan Chunlu [mailto:springri...@gmail.com] Sent: Saturday, July 23, 2011 9:16 AM To: cassandra-u...@incubator.apache.org Subject: do I need to add more nodes? minor compaction eat all IO I have three nodes and RF=3, every time it is do minor compaction, the cpu load(8 core) get to 30, and iostat -x 2 shows utils is 100%, is that means I need more nodes? the total data size is 60G thanks! --
Buildding a Ubuntu / Debian package for Cassandra
Guys, I am trying to build a debian package in order to deploy Cassandra 0.6.5 on Ubuntu. I see that you have a ./debian directory in the source builds, do you have a bit more background on how it is used and build? P.S. I am new to Ubuntu/Debian packaging so any type of pointer will help. Thanks, FR Francois Richard
RE: Buildding a Ubuntu / Debian package for Cassandra
Thanks Clint, I am going to look-up the links below, I am pretty new on the DEB packaging in general and from what I have seen so far, a lot of the tutorial on the web are mostly based on classic [ .configure | make | make install ] of an application built in C. In this case I wanted to figure out the DEB packaging in the context of a Java application. I'll read on more and will stay in touch. My goal at the end of the day, is to install the stock package for Cassandra and then to create a special Cassandra-config package that would move and deploy my customized configuration files on the system. Thanks, FR -Original Message- From: Clint Byrum [mailto:cl...@ubuntu.com] Sent: Thursday, September 16, 2010 10:54 AM To: user@cassandra.apache.org Subject: Re: Buildding a Ubuntu / Debian package for Cassandra Hello Francois, There are already .debs available here: http://wiki.apache.org/cassandra/DebianPackaging I've also setup a PPA to build the packages on Ubuntu here: https://launchpad.net/~cassandra-ubuntu/+archive/stable Its currently still at v0.6.4, but I am in the process of uploading 0.6.5 as I write this email.. The .debs are nearly identical. The only difference is that I've packaged the jars necessary to build, so that you get the same exact versions of all libraries if you need to patch + repeat the build. Also, these are built specifically for Ubuntu releases, so if we find any incompatibilities between debian/ubuntu we can fix them for ubuntu users. I hope this helps! On Sep 16, 2010, at 10:30 AM, Francois Richard wrote: Guys, I am trying to build a debian package in order to deploy Cassandra 0.6.5 on Ubuntu. I see that you have a ./debian directory in the source builds, do you have a bit more background on how it is used and build? P.S. I am new to Ubuntu/Debian packaging so any type of pointer will help. Thanks, FR Francois Richard
