If you are talking about 1.2.x then I also have memory problems on the
idle cluster: java memory constantly slow grows up to limit, then spend
long time for GC. I never seen such behaviour for 1.0.x and 1.1.x, where
on idle cluster java memory stay on the same value.
On 06/07/2013 05:19 PM, Joel Samuelsson wrote:
I keep having issues with GC. Besides the cluster mentioned above, we
also have a single node development cluster having the same issues.
This node has 12.33 GB data, a couple of million skinny rows and
basically no load. It has default memory settings but keep getting
very long stop-the-world GC pauses:
INFO [ScheduledTasks:1] 2013-06-07 10:37:02,537 GCInspector.java (line
122) GC for ParNew: 99342 ms for 1 collections, 1400754488 used; max
is 4114612224
To try to rule out amount of memory, I set it to 16GB (we're on a
virtual environment), with 4GB of it for Cassandra heap but that
didn't help either, the incredibly long GC pauses keep coming.
So I think something else is causing these issues, unless everyone is
having really long GC pauses (which I doubt). I came across this thread:
http://www.mail-archive.com/user@cassandra.apache.org/msg24042.html
suggesting # date -s “`date`” might help my issues. It didn't however
(unless I am supposed to replace that second date with the actual date?).
Has anyone had similar issues?
2013/4/17 aaron morton <aa...@thelastpickle.com
<mailto:aa...@thelastpickle.com>>
> INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java
(line 122) GC for ParNew: 338798 ms for 1 collections, 592212416
used; max is 1046937600
This does not say that the heap is full.
ParNew is GC activity for the new heap, which is typically a
smaller part of the overall heap.
It sounds like you are running with defaults for the memory
config, which is generally a good idea. But 4GB total memory for a
node is on the small size.
Try some changes, edit the cassandra-env.sh file and change
MAX_HEAP_SIZE="2G"
HEAP_NEWSIZE="400M"
You may also want to try:
MAX_HEAP_SIZE="2G"
HEAP_NEWSIZE="800M"
JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=4"
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=2"
The size of the new heap generally depends on the number of cores
available, see the commends in the -env file.
An older discussion about memory use, not that in 1.2 the bloom
filters (and compression data) are off heap now.
http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 17/04/2013, at 11:06 PM, Joel Samuelsson
<samuelsson.j...@gmail.com <mailto:samuelsson.j...@gmail.com>> wrote:
> You're right, it's probably hard. I should have provided more data.
>
> I'm running Ubuntu 10.04 LTS with JNA installed. I believe this
line in the log indicates that JNA is working, please correct me
if I'm wrong:
> CLibrary.java (line 111) JNA mlockall successful
>
> Total amount of RAM is 4GB.
>
> My description of data size was very bad. Sorry about that. Data
set size is 12.3 GB per node, compressed.
>
> Heap size is 998.44MB according to nodetool info.
> Key cache is 49MB bytes according to nodetool info.
> Row cache size is 0 bytes acoording to nodetool info.
> Max new heap is 205MB kbytes according to Memory Pool "Par Eden
Space" max in jconsole.
> Memtable is left at default which should give it 333MB according
to documentation (uncertain where I can verify this).
>
> Our production cluster seems similar to your dev cluster so
possibly increasing the heap to 2GB might help our issues.
>
> I am still interested in getting rough estimates of how much
heap will be needed as data grows. Other than empirical studies
how would I go about getting such estimates?
>
>
> 2013/4/16 Viktor Jevdokimov <viktor.jevdoki...@adform.com
<mailto:viktor.jevdoki...@adform.com>>
> How one could provide any help without any knowledge about your
cluster, node and environment settings?
>
>
>
> 40GB was calculated from 2 nodes with RF=2 (each has 100% data
range), 2.4-2.5M rows * 6 cols * 3kB as a minimum without
compression and any overhead (sstable, bloom filters and indexes).
>
>
>
> With ParNew GC time such as yours even if it is a swapping issue
I could say only that heap size is too small.
>
>
>
> Check Heap, New Heap sizes, memtable and cache sizes. Are you on
Linux? Is JNA installed and used? What is total amount of RAM?
>
>
>
> Just for a DEV environment we use 3 virtual machines with 4GB
RAM and use 2GB heap without any GC issue with amount of data from
0 to 16GB compressed on each node. Memtable space sized to 100MB,
New Heap 400MB.
>
>
>
> Best regards / Pagarbiai
> Viktor Jevdokimov
> Senior Developer
>
> Email: viktor.jevdoki...@adform.com
<mailto:viktor.jevdoki...@adform.com>
> Phone: +370 5 212 3063 <tel:%2B370%205%20212%203063>, Fax +370 5
261 0453 <tel:%2B370%205%20261%200453>
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider
> Take a ride with Adform's Rich Media Suite
> <signature-logo29.png>
> <signature-best-employer-logo4823.png>
>
> Disclaimer: The information contained in this message and
attachments is intended solely for the attention and use of the
named addressee and may be confidential. If you are not the
intended recipient, you are reminded that the information remains
the property of the sender. You must not use, disclose,
distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender
immediately and irrevocably delete this message and any copies.
>
> From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com
<mailto:samuelsson.j...@gmail.com>]
> Sent: Tuesday, April 16, 2013 12:52
> To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Subject: Re: Reduce Cassandra GC
>
>
>
> How do you calculate the heap / data size ratio? Is this a
linear ratio?
>
>
>
> Each node has slightly more than 12 GB right now though.
>
>
>
> 2013/4/16 Viktor Jevdokimov <viktor.jevdoki...@adform.com
<mailto:viktor.jevdoki...@adform.com>>
>
> For a >40GB of data 1GB of heap is too low.
>
>
>
> Best regards / Pagarbiai
>
> Viktor Jevdokimov
>
> Senior Developer
>
>
>
> Email: viktor.jevdoki...@adform.com
<mailto:viktor.jevdoki...@adform.com>
>
> Phone: +370 5 212 3063 <tel:%2B370%205%20212%203063>, Fax +370 5
261 0453 <tel:%2B370%205%20261%200453>
>
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
>
> Follow us on Twitter: @adforminsider
>
> Take a ride with Adform's Rich Media Suite
>
> <image001.png>
>
> <image002.png>
>
>
> Disclaimer: The information contained in this message and
attachments is intended solely for the attention and use of the
named addressee and may be confidential. If you are not the
intended recipient, you are reminded that the information remains
the property of the sender. You must not use, disclose,
distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender
immediately and irrevocably delete this message and any copies.
>
>
>
> From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com
<mailto:samuelsson.j...@gmail.com>]
> Sent: Tuesday, April 16, 2013 10:47
> To: user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Subject: Reduce Cassandra GC
>
>
>
> Hi,
>
>
>
> We have a small production cluster with two nodes. The load on
the nodes is very small, around 20 reads / sec and about the same
for writes. There are around 2.5 million keys in the cluster and a
RF of 2.
>
>
>
> About 2.4 million of the rows are skinny (6 columns) and around
3kb in size (each). Currently, scripts are running, accessing all
of the keys in timeorder to do some calculations.
>
>
>
> While running the scripts, the nodes go down and then come back
up 6-7 minutes later. This seems to be due to GC. I get lines like
this in the log:
>
> INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java
(line 122) GC for ParNew: 338798 ms for 1 collections, 592212416
used; max is 1046937600
>
>
>
> However, the heap is not full. The heap usage has a jagged
pattern going from 60% up to 70% during 5 minutes and then back
down to 60% the next 5 minutes and so on. I get no "Heap is X
full..." messages. Every once in a while at one of these peaks, I
get these stop-the-world GC for 6-7 minutes. Why does GC take up
so much time even though the heap isn't full?
>
>
>
> I am aware that my access patterns make key caching very
unlikely to be high. And indeed, my average key cache hit ratio
during the run of the scripts is around 0.5%. I tried disabling
key caching on the accessed column family (UPDATE COLUMN FAMILY cf
WITH caching=none;) through the cassandra-cli but I get the same
behaviour. Is the turning key cache off effective immediately?
>
>
>
> Stop-the-world GC is fine if it happens for a few seconds but
having them for several minutes doesn't work. Any other
suggestions to remove them?
>
>
>
> Best regards,
>
> Joel Samuelsson
>
>
>
>
