Re: Unbalanced disk load

[J. Ryan Earl]

Even with https://issues.apache.org/jira/browse/CASSANDRA-7386 data
balancing across JBOD setups is pretty horrible.  Having used JBOD for
about 2 years from 1.2.x and up, it is my opinion JBOD on Cassandra is
nascent at best and far from mature.  For a variety of reasons, JBOD should
perform better if IO and data is balanced across multiple devices due to
things like linux device queues, striping overhead, access contention, and
so forth.  However, data and access patterns simply are not balanced in
Cassandra JBOD setups.

Here's an example of what we see on one of our nodes:

/dev/sdd1 1.1T  202G  915G  19% /data/2
/dev/sde1 1.1T  136G  982G  13% /data/3
/dev/sdi1 1.1T  217G  901G  20% /data/7
/dev/sdc1 1.1T  402G  715G  36% /data/1
/dev/sdh1 1.1T  187G  931G  17% /data/6
/dev/sdf1 1.1T  201G  917G  18% /data/4
/dev/sdg1 1.1T  154G  963G  14% /data/5

Essentially, for a storage engine to make good use of JBOD, like HDFS or
Ceph does, the storage engines need to essentially be designed from the
ground up to use JBOD.  In Cassandra, a single sstable cannot be split at
the storage engine level to be split across members of the JBOD.  In our
case, we have a single sstable file that is bigger than all the data files
combined on other disks.  Looking at the disk using 402G, we see:

274G vcells_polished_data-vcells_polished-jb-38985-Data.db

A single sstable is using 274G.  In addition to data usage imbalance, we
see hot spots as well.  With static fields in particular, and CFs that
don't change much, you'll get CFs that end up compacting into fewer number
of large sstables.  With most of the data for a CF being in one sstable and
on one data volume, a single data volume then becomes a hotspot for reads
on that CF.  Cassandra tries to minimize the number of sstables a row will
be written across, but in particular after some compaction on an CFs that
are rarely updated, most of the data for a CF can end up in a single
sstable, and stables aren't split aross data volumes.  Thus a single volume
will be a hot-spot for access to that CF in a JBOD setup as Cassandra does
not effectively distribute data across individual volumes in all
circumstances.

There may be tuning which would help this, but it's specific to JBOD and
not somewhat that you would have to worry about in a single data volume
setup, ie RAID0.  With a RAID0, the downside of course, is that losing a
single member disk to the RAID0 takes the node down.  The upside is you
don't have to worry about the imbalance of both I/O and data footprint
across individual volumes.

Unlike HDFS, Ceph, and RAID for that matter, where you're dealing with
maximum fixed sizes blocks/stripes that are then distributed at a granular
level across the JBOD volumes, Cassandra is dealing with uncapped, low
granularity, variable sized sstable data files which it attempts to
distribute across JBOD volumes making JBOD far from ideal.  Frankly, it's
hard for me to imagine any columnar data store doing JBOD well.

On Fri, Jul 17, 2015 at 4:08 PM, Soerian Lieve  wrote:

> Hi,
>
> I am currently benchmarking Cassandra with three machines, and on each
> machine I am seeing an unbalanced distribution of data among the data
> directories (1 per disk).
> I am concerned that this affects my write performance, is there anything
> that I can make the distribution be more even? Would raid0 be my best
> option?
>
> Details:
> 3 machines, each have 24 cores, 64GB of RAM, 7 SSDs of 500GB each.
> Commitlog is on a separate disk, cassandra.yaml configured according to
> Datastax' guide on cassandra.yaml.
> Total size of data is about 2TB, 14B records, all unique. Replication
> factor of 1.
>
> Thanks,
> Soerian
>

Re: error='Cannot allocate memory' (errno=12)

[J. Ryan Earl]

I see your ulimit -a above, missed that.  You should increase nofile
ulimit.  If you used JNA, you'd need to increase memlock too, but you
probably aren't using JNA.  1024 nofile is default and far too small, try
making that like 64K.  Thread handles can count against file descriptors
limits, similar to how pipes, sockets, etc can count against file
descriptor limits.

On Tue, May 12, 2015 at 7:55 PM, J. Ryan Earl  wrote:

> What's your ulimit -a output?  Did you adjust nproc and nofile ulimits
> up?  Do you have JNA installed?  What about memlock ulimit and in
> sysctl.conf => kernel.shmmax?
>
> What's in cassandra.log?
>
> On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj <
> rahul.bhard...@indiamart.com> wrote:
>
>> Hi Robert,
>>
>> I saw somewhere you answering the same prob but no solution found. plz
>> check again.
>>
>>
>> regards:
>> Rahul Bhardwaj
>>
>>
>>
>>
>> On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj <
>> rahul.bhard...@indiamart.com> wrote:
>>
>>> Bu it is giving same error with java 7 and open jdk
>>>
>>> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal 
>>> wrote:
>>>
>>>> Well i havent used 2.1.x cassandra neither java 8 but any reason for
>>>> not using oracle JDK as i thought thats what is recommended. i saw a thread
>>>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
>>>> 2.1.x versions.
>>>>
>>>>
>>>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
>>>> rahul.bhard...@indiamart.com> wrote:
>>>>
>>>>> PFA of error log​
>>>>>  hs_err_pid9656.log
>>>>> <https://docs.google.com/a/indiamart.com/file/d/0B0hlSlesIPVfaU9peGwxSXdsZGc/edit?usp=drive_web>
>>>>> ​
>>>>>
>>>>> On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
>>>>> rahul.bhard...@indiamart.com> wrote:
>>>>>
>>>>>> free RAM:
>>>>>>
>>>>>>
>>>>>> free -m
>>>>>>  total   used   free sharedbuffers
>>>>>> cached
>>>>>> Mem: 64398  23753  40644  0108
>>>>>> 8324
>>>>>> -/+ buffers/cache:  15319  49078
>>>>>> Swap: 2925 15   2909
>>>>>>
>>>>>>
>>>>>>  ulimit -a
>>>>>> core file size  (blocks, -c) 0
>>>>>> data seg size   (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size   (blocks, -f) unlimited
>>>>>> pending signals (-i) 515041
>>>>>> max locked memory   (kbytes, -l) 64
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files  (-n) 1024
>>>>>> pipe size(512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>> real-time priority  (-r) 0
>>>>>> stack size  (kbytes, -s) 10240
>>>>>> cpu time   (seconds, -t) unlimited
>>>>>> max user processes  (-u) 515041
>>>>>> virtual memory  (kbytes, -v) unlimited
>>>>>> file locks  (-x) unlimited
>>>>>>
>>>>>>
>>>>>> Also attaching complete error file
>>>>>>
>>>>>>
>>>>>> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal 
>>>>>> wrote:
>>>>>>
>>>>>>> the memory cassandra is trying to allocate is pretty small. you sure
>>>>>>> there is no hardware failure on the machine. what is the free ram on the
>>>>>>> box ?
>>>>>>>
>>>>>>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>>>>>>> rahul.bhard...@indiamart.com> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> We have cluster of 3 nodes with  64GB RAM each. My cluster was
>>>>>>>> running in healthy state. Suddenly one machine's cassandra daemon stops
>>>>>>>> working and shut down.
>>>>>>>>
>>>>

Re: error='Cannot allocate memory' (errno=12)

[J. Ryan Earl]

What's your ulimit -a output?  Did you adjust nproc and nofile ulimits up?
Do you have JNA installed?  What about memlock ulimit and in sysctl.conf =>
kernel.shmmax?

What's in cassandra.log?

On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj <
rahul.bhard...@indiamart.com> wrote:

> Hi Robert,
>
> I saw somewhere you answering the same prob but no solution found. plz
> check again.
>
>
> regards:
> Rahul Bhardwaj
>
>
>
>
> On Mon, May 11, 2015 at 5:49 PM, Rahul Bhardwaj <
> rahul.bhard...@indiamart.com> wrote:
>
>> Bu it is giving same error with java 7 and open jdk
>>
>> On Mon, May 11, 2015 at 5:26 PM, Anishek Agarwal 
>> wrote:
>>
>>> Well i havent used 2.1.x cassandra neither java 8 but any reason for not
>>> using oracle JDK as i thought thats what is recommended. i saw a thread
>>> earlier stating java 8 with 2.0.14+ cassandra is tested but not sure about
>>> 2.1.x versions.
>>>
>>>
>>> On Mon, May 11, 2015 at 4:04 PM, Rahul Bhardwaj <
>>> rahul.bhard...@indiamart.com> wrote:
>>>
 PFA of error log​
  hs_err_pid9656.log
 
 ​

 On Mon, May 11, 2015 at 3:58 PM, Rahul Bhardwaj <
 rahul.bhard...@indiamart.com> wrote:

> free RAM:
>
>
> free -m
>  total   used   free sharedbuffers
> cached
> Mem: 64398  23753  40644  0108
> 8324
> -/+ buffers/cache:  15319  49078
> Swap: 2925 15   2909
>
>
>  ulimit -a
> core file size  (blocks, -c) 0
> data seg size   (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size   (blocks, -f) unlimited
> pending signals (-i) 515041
> max locked memory   (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files  (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority  (-r) 0
> stack size  (kbytes, -s) 10240
> cpu time   (seconds, -t) unlimited
> max user processes  (-u) 515041
> virtual memory  (kbytes, -v) unlimited
> file locks  (-x) unlimited
>
>
> Also attaching complete error file
>
>
> On Mon, May 11, 2015 at 3:35 PM, Anishek Agarwal 
> wrote:
>
>> the memory cassandra is trying to allocate is pretty small. you sure
>> there is no hardware failure on the machine. what is the free ram on the
>> box ?
>>
>> On Mon, May 11, 2015 at 3:28 PM, Rahul Bhardwaj <
>> rahul.bhard...@indiamart.com> wrote:
>>
>>> Hi All,
>>>
>>> We have cluster of 3 nodes with  64GB RAM each. My cluster was
>>> running in healthy state. Suddenly one machine's cassandra daemon stops
>>> working and shut down.
>>>
>>> On restarting it after 2 minutes it again stops and is getting stop
>>> after returning below error in cassandra.log
>>>
>>> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>> os::commit_memory(0x7fd064dc6000, 12288, 0) failed; error='Cannot
>>> allocate memory' (errno=12)
>>> #
>>> # There is insufficient memory for the Java Runtime Environment to
>>> continue.
>>> # Native memory allocation (malloc) failed to allocate 12288 bytes
>>> for committing reserved memory.
>>> # An error report file with more information is saved as:
>>> # /tmp/hs_err_pid23215.log
>>> INFO  09:50:41 Loading settings from
>>> file:/etc/cassandra/default.conf/cassandra.yaml
>>> INFO  09:50:41 Node
>>> configuration:[authenticator=AllowAllAuthenticator;
>>> authorizer=AllowAllAuthorizer; auto_snapshot=true;
>>> batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
>>> cas_contention_timeout_in_ms=1000; client_encryption_options=;
>>> cluster_name=Test Cluster; column_index_size_in_kb=64;
>>> commit_failure_policy=stop;
>>> commitlog_directory=/var/lib/cassandra/commitlog;
>>> commitlog_segment_size_in_mb=64; commitlog_sync=periodic;
>>> commitlog_sync_period_in_ms=1; compaction_throughput_mb_per_sec=16;
>>> concurrent_compactors=4; concurrent_counter_writes=32; 
>>> concurrent_reads=32;
>>> concurrent_writes=32; counter_cache_save_period=7200;
>>> counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
>>> cross_node_timeout=false; 
>>> data_file_directories=[/var/lib/cassandra/data];
>>> disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1;
>>> dynamic_snitch_reset_interval_in_ms=60;
>>> dynamic_snitch_update_interval_in_ms=100;
>>> endpoint_snitch=GossipingPropertyFileSnitch; 
>>> hinted_handoff_enabled=true;
>>> hinted_handoff_throttle_in_kb=1024; incremental_b

Re: OOM and high SSTables count

[J. Ryan Earl]

We think it is this bug:
https://issues.apache.org/jira/browse/CASSANDRA-8860

We're rolling a patch to beta before rolling it into production.

On Wed, Mar 4, 2015 at 4:12 PM, graham sanderson  wrote:

> We can confirm a problem on 2.1.3 (sadly our beta sstable state obviously
> did not match our production ones in some critical way)
>
> We have about 20k sstables on each of 6 nodes right now; actually a quick
> glance shows 15k of those are from OpsCenter, which may have something to
> do with beta/production mismatch
>
> I will look into the open OOM JIRA issue against 2.1.3 - we may being
> penalized for heavy use of JBOD (x7 per node)
>
> It also looks like 2.1.3 is leaking memory, though it eventually recovers
> via GCInspector causing a complete memtable flush.
>
> On Mar 4, 2015, at 12:31 PM, daemeon reiydelle  wrote:
>
> Are you finding a correlation between the shards on the OOM DC1 nodes and
> the OOM DC2 nodes? Does your monitoring tool indicate that the DC1 nodes
> are using significantly more CPU (and memory) than the nodes that are NOT
> failing? I am leading you down the path to suspect that your sharding is
> giving you hot spots. Also are you using vnodes?
>
> Patrick
>
>>
>> On Wed, Mar 4, 2015 at 9:26 AM, Jan  wrote:
>>
>>> HI Roni;
>>>
>>> You mentioned:
>>> DC1 servers have 32GB of RAM and 10GB of HEAP. DC2 machines have 16GB of
>>> RAM and 5GB HEAP.
>>>
>>> Best practices would be be to:
>>> a)  have a consistent type of node across both DC's.  (CPUs, Memory,
>>> Heap & Disk)
>>> b)  increase heap on DC2 servers to be  8GB for C* Heap
>>>
>>> The leveled compaction issue is not addressed by this.
>>> hope this helps
>>>
>>> Jan/
>>>
>>>
>>>
>>>
>>>   On Wednesday, March 4, 2015 8:41 AM, Roni Balthazar <
>>> ronibaltha...@gmail.com> wrote:
>>>
>>>
>>> Hi there,
>>>
>>> We are running C* 2.1.3 cluster with 2 DataCenters: DC1: 30 Servers /
>>> DC2 - 10 Servers.
>>> DC1 servers have 32GB of RAM and 10GB of HEAP. DC2 machines have 16GB
>>> of RAM and 5GB HEAP.
>>> DC1 nodes have about 1.4TB of data and DC2 nodes 2.3TB.
>>> DC2 is used only for backup purposes. There are no reads on DC2.
>>> Every writes and reads are on DC1 using LOCAL_ONE and the RF DC1: 2 and
>>> DC2: 1.
>>> All keyspaces have STCS (Average 20~30 SSTables count each table on
>>> both DCs) except one that is using LCS (DC1: Avg 4K~7K SSTables / DC2:
>>> Avg 3K~14K SSTables).
>>>
>>> Basically we are running into 2 problems:
>>>
>>> 1) High SSTables count on keyspace using LCS (This KS has 500GB~600GB
>>> of data on each DC1 node).
>>> 2) There are 2 servers on DC1 and 4 servers in DC2 that went down with
>>> the OOM error message below:
>>>
>>> ERROR [SharedPool-Worker-111] 2015-03-04 05:03:26,394
>>> JVMStabilityInspector.java:94 - JVM state determined to be unstable.
>>> Exiting forcefully due to:
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.cassandra.db.composites.CompoundSparseCellNameType.copyAndMakeWith(CompoundSparseCellNameType.java:186)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.composites.AbstractCompoundCellNameType$CompositeDeserializer.readNext(AbstractCompoundCellNameType.java:286)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.AtomDeserializer.readNext(AtomDeserializer.java:104)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:426)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:350)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:142)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:44)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>>> ~[guava-16.0.jar:na]
>>> at
>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>>> ~[guava-16.0.jar:na]
>>> at
>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:172)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:155)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
>>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>> at
>>> org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125)
>>> ~[apache-cas

Re: Setting up JNA on CentOS 6.6. with cassandra20-2.0.12 and Oracle Java 1.7.0_75

[J. Ryan Earl]

CentOS6 and every major version of C* from 1.1 through 2.1, but I would be
curious if there's maybe a memory leak or something fixed between 3.2.4 and
3.2.7...?  AFAIK, it's only use for memlocking the heap and creating
hardlinks for snapshots, both of which work.

On Wed, Feb 25, 2015 at 2:53 PM, Garret Pick  wrote:

> Hi,
>
> On this page
>
>
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installJnaRHEL.html
>
> it says
>
> "Cassandra requires JNA 3.2.7 or later. Some Yum repositories may provide
> earlier versions"
>
> and at the bottom
>
> "If you can't install using Yum or it provides a version of the JNA
> earlier than 3.2.7, install as described in Installing the JNA from the JAR
> file."
>
> Which version of OS and Cassandra are you running?
>
> thanks,
> Garret
>
>
>
>
> On Wed, Feb 25, 2015 at 10:46 AM, J. Ryan Earl  wrote:
> >
> > We've been using jna-3.2.4-2.el6.x86_64 with the Sun/Oracle JDK for
> probably 2-years now, and it works just fine.  Where are you seeing 3.2.7
> required at?  I searched the pages you link and that string isn't even in
> there.
> >
> > Regardless, I assure you the newest jna that ships in the EL6 repo works
> without issues.
> >
> > On Wed, Feb 25, 2015 at 2:12 PM, Garret Pick  wrote:
> >>
> >> Hello,
> >>
> >> I'm having problems getting cassandra to start with the configuration
> listed above.
> >>
> >> Yum wants to install 3.2.4-2.el6 of the JNA along with several other
> packages including java-1.7.0-openjdk
> >>
> >> The documentation states that a JNA version earlier that 3.2.7 should
> not be used, so the jar file should be downloaded and installed directly
> into C*'s lib directory per
> >>
> >>
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installJnaTar.html
> >>
> >> From /var/log/cassandra/system.log
> >>
> >> all I see is
> >>
> >>  INFO [main] 2015-02-25 20:06:10,202 CassandraDaemon.java (line 191)
> Classpath:
> /etc/cassandra/conf:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.12.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.12.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.12.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
> >>
> >> and it never actually starts
> >>
> >> Note that JNA is in the classpath above and is when I remove it,
> cassandra starts successfully.
> >>
> >> I tried installing the DSE package and it looks like it wants to
> install the older 3.2.4 JNA as a dependency so there seems to be a
> discrepancy in documentation
> >>
> >> Per
> >>
> >>
> http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installRHELdse.html
> >>
> >> Note: JNA (Java Native Access) is automatically installed.
> >>
> >> thanks for any help,
> >> Garret
> >
> >
>

Re: Setting up JNA on CentOS 6.6. with cassandra20-2.0.12 and Oracle Java 1.7.0_75

[J. Ryan Earl]

We've been using jna-3.2.4-2.el6.x86_64 with the Sun/Oracle JDK for
probably 2-years now, and it works just fine.  Where are you seeing 3.2.7
required at?  I searched the pages you link and that string isn't even in
there.

Regardless, I assure you the newest jna that ships in the EL6 repo works
without issues.

On Wed, Feb 25, 2015 at 2:12 PM, Garret Pick  wrote:

> Hello,
>
> I'm having problems getting cassandra to start with the configuration
> listed above.
>
> Yum wants to install 3.2.4-2.el6 of the JNA along with several other
> packages including java-1.7.0-openjdk
>
> The documentation states that a JNA version earlier that 3.2.7 should not
> be used, so the jar file should be downloaded and installed directly into
> C*'s lib directory per
>
>
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installJnaTar.html
>
> From /var/log/cassandra/system.log
>
> all I see is
>
>  INFO [main] 2015-02-25 20:06:10,202 CassandraDaemon.java (line 191)
> Classpath:
> /etc/cassandra/conf:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.12.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.12.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.12.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
>
> and it never actually starts
>
> Note that JNA is in the classpath above and is when I remove it, cassandra
> starts successfully.
>
> I tried installing the DSE package and it looks like it wants to install
> the older 3.2.4 JNA as a dependency so there seems to be a discrepancy in
> documentation
>
> Per
>
>
> http://www.datastax.com/documentation/datastax_enterprise/4.6/datastax_enterprise/install/installRHELdse.html
>
> Note: JNA (Java Native Access) is automatically installed.
>
> thanks for any help,
> Garret
>

Re: ava.lang.OutOfMemoryError: unable to create new native thread

[J. Ryan Earl]

What's the 'ulimit -a' output of the user cassandra runs as?  From this and
your previous OOM thread, is sounds like you skipped the requisite OS
configuration.

On Wed, Sep 17, 2014 at 9:43 AM, Yatong Zhang  wrote:

> Hi there,
>
> I am using leveled compaction strategy and have many sstable files. The
> error was during the startup, so any idea about this?
>
>
>> ERROR [FlushWriter:4] 2014-09-17 22:36:59,383 CassandraDaemon.java (line
>> 199) Exception in thread Thread[FlushWriter:4,5,main]
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:693)
>> at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> at
>> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> ERROR [FlushWriter:2] 2014-09-17 22:36:59,472 CassandraDaemon.java (line
>> 199) Exception in thread Thread[FlushWriter:2,5,main]
>> FSReadError in
>> /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-23-Index.db
>> at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
>> at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:334)
>> at
>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:324)
>> at
>> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:394)
>> at
>> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:342)
>> at
>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>> at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> Caused by: java.io.IOException: Map failed
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
>> at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
>> ... 10 more
>> Caused by: java.lang.OutOfMemoryError: Map failed
>> at sun.nio.ch.FileChannelImpl.map0(Native Method)
>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846)
>> ... 11 more
>>
>
>

Re: hs_err_pid3013.log, out of memory?

[J. Ryan Earl]

On Wed, Sep 17, 2014 at 8:35 PM, Yatong Zhang  wrote:

> @Chris Lohfink I have 16G memory per node, all the other settings are
> default
>
> @J. Ryan Earl I am not sure. I am using the default settings.
>
> But I've found out it might be because some settings in
> '/etc/sysctl.conf'. I am still testing it
>


If JNA is installed, it will try to memlock all of the JVM process.  For
this to happen, you have to adjust the settings for the user you run
Cassandra as under /etc/security/limits.conf or limits.d/ and you have to
modifying kernel.shmmax in sysctl.conf accordingly.  If you do not, and JNA
is installed, the memlock will fail with the error you gave.

Re: hs_err_pid3013.log, out of memory?

[J. Ryan Earl]

Are you using JNA?  Did you adjust your memlock limit?

On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink 
wrote:

> How much memory does your system have? How much memory is system utilizing
> before starting Cassandra (use command free)? What are the heap setting it
> tries to use?
>
> Chris
>
> On Sep 15, 2014, at 8:16 PM, Yatong Zhang  wrote:
>
> It's during the startup. I tried to upgrade cassandra from 2.0.7 to
> 2.0.10, but looks like cassandra could not start again. Also I found the
> following log at '/var/log/messages':
>
> Sep 16 09:06:59 storage6 kernel: INFO: task java:4971 blocked for more
>> than 120 seconds.
>> Sep 16 09:06:59 storage6 kernel:  Tainted: G
>> --- H  2.6.32-431.el6.x86_64 #1
>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Sep 16 09:06:59 storage6 kernel: java  D 0003 0
>> 4971  1 0x0080
>> Sep 16 09:06:59 storage6 kernel: 88042b591c98 0082
>> 81ed4ff0 8803b4f01540
>> Sep 16 09:06:59 storage6 kernel: 88042b591c68 810af370
>> 88042b591ca0 8803b4f01540
>> Sep 16 09:06:59 storage6 kernel: 8803b4f01af8 88042b591fd8
>> fbc8 8803b4f01af8
>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> exit_robust_list+0x90/0x160
>> Sep 16 09:06:59 storage6 kernel: [] exit_mm+0x95/0x180
>> Sep 16 09:06:59 storage6 kernel: [] do_exit+0x15f/0x870
>> Sep 16 09:06:59 storage6 kernel: []
>> do_group_exit+0x58/0xd0
>> Sep 16 09:06:59 storage6 kernel: []
>> get_signal_to_deliver+0x1f6/0x460
>> Sep 16 09:06:59 storage6 kernel: [] do_signal+0x75/0x800
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> wake_up_new_task+0xd9/0x130
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> do_fork+0x13d/0x480
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> sys_futex+0x7b/0x170
>> Sep 16 09:06:59 storage6 kernel: []
>> do_notify_resume+0x90/0xc0
>> Sep 16 09:06:59 storage6 kernel: [] int_signal+0x12/0x17
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4972 blocked for more
>> than 120 seconds.
>> Sep 16 09:06:59 storage6 kernel:  Tainted: G
>> --- H  2.6.32-431.el6.x86_64 #1
>> Sep 16 09:06:59 storage6 kernel: "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Sep 16 09:06:59 storage6 kernel: java  D  0
>> 4972  1 0x0080
>> Sep 16 09:06:59 storage6 kernel: 8803b4d7fc98 0082
>> 81ed6d78 8803b4cf1500
>> Sep 16 09:06:59 storage6 kernel: 8803b4d7fc68 810af370
>> 8803b4d7fca0 8803b4cf1500
>> Sep 16 09:06:59 storage6 kernel: 8803b4cf1ab8 8803b4d7ffd8
>> fbc8 8803b4cf1ab8
>> Sep 16 09:06:59 storage6 kernel: Call Trace:
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> exit_robust_list+0x90/0x160
>> Sep 16 09:06:59 storage6 kernel: [] exit_mm+0x95/0x180
>> Sep 16 09:06:59 storage6 kernel: [] do_exit+0x15f/0x870
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> wake_up_state+0x10/0x20
>> Sep 16 09:06:59 storage6 kernel: []
>> do_group_exit+0x58/0xd0
>> Sep 16 09:06:59 storage6 kernel: []
>> get_signal_to_deliver+0x1f6/0x460
>> Sep 16 09:06:59 storage6 kernel: [] do_signal+0x75/0x800
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> __switch_to+0x1ac/0x320
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> thread_return+0x4e/0x76e
>> Sep 16 09:06:59 storage6 kernel: [] ?
>> sys_futex+0x7b/0x170
>> Sep 16 09:06:59 storage6 kernel: []
>> do_notify_resume+0x90/0xc0
>> Sep 16 09:06:59 storage6 kernel: [] int_signal+0x12/0x17
>> Sep 16 09:06:59 storage6 kernel: INFO: task java:4973 blocked for more
>> than 120 seconds.
>>
>
>
> On Tue, Sep 16, 2014 at 9:00 AM, Robert Coli  wrote:
>
>> On Mon, Sep 15, 2014 at 5:55 PM, Yatong Zhang 
>> wrote:
>>
>>> I just encountered an error which left a log '/hs_err_pid3013.log'. So
>>> is there a way to solve this?
>>>
>>> # There is insufficient memory for the Java Runtime Environment to
 continue.
 # Native memory allocation (malloc) failed to allocate 12288 bytes for
 committing reserved memory.

>>>
>> Use less heap memory?
>>
>> You haven't specified under which circumstances this occurred, so I can
>> only conjecture that it is likely being caused by "writing too fast".
>>
>> Write more slowly.
>>
>> =Rob
>>
>>
>
>

Re: too many open files

[J. Ryan Earl]

Yes, definitely look how many open files are actual file handles versus
networks sockets.  We found a file handle leak in 2.0 but it was patched in
2.0.3 or .5 I think.  A million open files is way too high.


On Fri, Aug 8, 2014 at 5:19 PM, Andrey Ilinykh  wrote:

> You may have this problem if your client doesn't reuse the connection but
> opens new every type. So, run netstat and check the number of established
> connections. This number should not be big.
>
> Thank you,
>   Andrey
>
>
> On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <
> marc...@s1mbi0se.com.br> wrote:
>
>> Hi,
>>
>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too
>> many open files" exceptions when I try to perform a large number of
>> operations in my 10 node cluster.
>>
>> I saw the documentation
>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>> and I have set everything to the recommended settings, but I keep getting
>> the errors.
>>
>> In the documentation it says: "Another, much less likely possibility, is
>> a file descriptor leak in Cassandra. Run lsof -n | grep java to check
>> that the number of file descriptors opened by Java is reasonable and
>> reports the error if the number is greater than a few thousand."
>>
>> I guess it's not the case, or else a lot of people would be complaining
>> about it, but I am not sure what I could do to solve the problem.
>>
>> Any hint about how to solve it?
>>
>> My client is written in python and uses Cassandra Python Driver. Here are
>> the exceptions I am having in the client:
>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.144
>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.77
>>
>>
>> And here is the exception I am having in the server:
>>
>>  WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>> BatchStatement.java (line 223) Batch of prepared statements for
>> [identification.entity_lookup, identification.entity] is of size 25216,
>> exceeding specified threshold of 5120 by 20096.
>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>> ErrorMessage.java (line 222) Unexpected exception during request
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>

Re: Write performance with 1.2.12

[J. Ryan Earl]

Why did you switch to RandomPartitioner away from Murmur3Partitioner?  Have
you tried with Murmur3?


   1. # partitioner: org.apache.cassandra.dht.Murmur3Partitioner
   2. partitioner: org.apache.cassandra.dht.RandomPartitioner



On Fri, Dec 6, 2013 at 10:36 AM, srmore  wrote:

>
>
>
> On Fri, Dec 6, 2013 at 9:59 AM, Vicky Kak  wrote:
>
>> You have passed the JVM configurations and not the cassandra
>> configurations which is in cassandra.yaml.
>>
>
> Apologies, was tuning JVM and that's what was in my mind.
> Here are the cassandra settings http://pastebin.com/uN42GgYT
>
>
>
>> The spikes are not that significant in our case and we are running the
>> cluster with 1.7 gb heap.
>>
>> Are these spikes causing any issue at your end?
>>
>
> There are no big spikes, the overall performance seems to be about 40% low.
>
>
>>
>>
>>
>>
>> On Fri, Dec 6, 2013 at 9:10 PM, srmore  wrote:
>>
>>>
>>>
>>>
>>> On Fri, Dec 6, 2013 at 9:32 AM, Vicky Kak  wrote:
>>>
 Hard to say much without knowing about the cassandra configurations.

>>>
>>> The cassandra configuration is
>>> -Xms8G
>>> -Xmx8G
>>> -Xmn800m
>>> -XX:+UseParNewGC
>>> -XX:+UseConcMarkSweepGC
>>> -XX:+CMSParallelRemarkEnabled
>>> -XX:SurvivorRatio=4
>>> -XX:MaxTenuringThreshold=2
>>> -XX:CMSInitiatingOccupancyFraction=75
>>> -XX:+UseCMSInitiatingOccupancyOnly
>>>
>>>
>>>
 Yes compactions/GC's could skipe the CPU, I had similar behavior with
 my setup.

>>>
>>> Were you able to get around it ?
>>>
>>>

 -VK


 On Fri, Dec 6, 2013 at 7:40 PM, srmore  wrote:

> We have a 3 node cluster running cassandra 1.2.12, they are pretty big
> machines 64G ram with 16 cores, cassandra heap is 8G.
>
> The interesting observation is that, when I send traffic to one node
> its performance is 2x more than when I send traffic to all the nodes. We
> ran 1.0.11 on the same box and we observed a slight dip but not half as
> seen with 1.2.12. In both the cases we were writing with LOCAL_QUORUM.
> Changing CL to ONE make a slight improvement but not much.
>
> The read_Repair_chance is 0.1. We see some compactions running.
>
> following is my iostat -x output, sda is the ssd (for commit log) and
> sdb is the spinner.
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>   66.460.008.950.010.00   24.58
>
> Device: rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz
> avgqu-sz   await  svctm  %util
> sda   0.0027.60  0.00  4.40 0.00   256.00
> 58.18 0.012.55   1.32   0.58
> sda1  0.00 0.00  0.00  0.00 0.00 0.00
> 0.00 0.000.00   0.00   0.00
> sda2  0.0027.60  0.00  4.40 0.00   256.00
> 58.18 0.012.55   1.32   0.58
> sdb   0.00 0.00  0.00  0.00 0.00 0.00
> 0.00 0.000.00   0.00   0.00
> sdb1  0.00 0.00  0.00  0.00 0.00 0.00
> 0.00 0.000.00   0.00   0.00
> dm-0  0.00 0.00  0.00  0.00 0.00 0.00
> 0.00 0.000.00   0.00   0.00
> dm-1  0.00 0.00  0.00  0.60 0.00 4.80
> 8.00 0.005.33   2.67   0.16
> dm-2  0.00 0.00  0.00  0.00 0.00 0.00
> 0.00 0.000.00   0.00   0.00
> dm-3  0.00 0.00  0.00 24.80 0.00   198.40
> 8.00 0.249.80   0.13   0.32
> dm-4  0.00 0.00  0.00  6.60 0.0052.80
> 8.00 0.011.36   0.55   0.36
> dm-5  0.00 0.00  0.00  0.00 0.00 0.00
> 0.00 0.000.00   0.00   0.00
> dm-6  0.00 0.00  0.00 24.80 0.00   198.40
> 8.00 0.29   11.60   0.13   0.32
>
>
>
> I can see I am cpu bound here but couldn't figure out exactly what is
> causing it, is this caused by GC or Compaction ? I am thinking it is
> compaction, I see a lot of context switches and interrupts in my vmstat
> output.
>
> I don't see GC activity in the logs but see some compaction activity.
> Has anyone seen this ? or know what can be done to free up the CPU.
>
> Thanks,
> Sandeep
>
>
>

>>>
>>
>

Re: cassandra performance problems

[J. Ryan Earl]

On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev wrote:

> We've plugged it into our production environment as a cache in front of
> postgres. Everything worked fine, we even stressed it by explicitly
> propagating about 30G (10G/node) data from postgres to cassandra.
>

If you just want a caching layer, why wouldn't you use Memcached or Redis
instead?  Cassandra is designed to be a persist store and not so much
designed as caching layer.  If you were replacing your use of Postgres
completely, that would be appropriate.

Re: Getting into Too many open files issues

[J. Ryan Earl]

There was a bug introduced in 2.0.0-beta1 related to TTL, a patch just came
available in: https://issues.apache.org/jira/browse/CASSANDRA-6275


On Thu, Nov 7, 2013 at 5:15 AM, Murthy Chelankuri wrote:

> I have experimenting cassandra latest version for storing the huge the in
> our application.
>
> Write are doing good. but when comes to reads i have obsereved that
> cassandra is getting into too many open files issues. When i check the logs
> its not able to open the cassandra data files any more before of the file
> descriptors limits.
>
>
> Can some one suggest me what i am going wrong what could be issues which
> causing the read operating leads to Too many open files issue.
>

Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled

[J. Ryan Earl]

First off, I'm curious what hardware (system specs) you're running this on?

Secondly, here are some observations:
* You're not running the newest JDK7, I can tell by your stack-size.
 Consider getting the newest.

* Cassandra 2.0.2 has a lot of improvements, consider upgrading.  We
noticed improved heap usage compared to 2.0.2

* Have you simply tried decreasing the size of your row cache?  Tried 256MB?

* Do you have JNA installed?  Otherwise, you're not getting off-heap usage
for these caches which seems likely.  Check your cassandra.log to verify
JNA operation.

* Your NewGen is too small.  See your heap peaks?  This is because
short-lived memory is being put into OldGen, which only gets cleaned up
during fullGC.  You should set your NewGen to about 25-30% of your total
heapsize.  Many objects are short-lived, and CMS GC is significantly more
efficient if the shorter-lived objects never get promoted to OldGen; you'll
get more concurrent, non-blocking GC.  If you're not using JNA (per above)
row-cache and key-cache is still on-heap, so you want your NewGen to be >=
twice as large as the size of these combined caches.  You should never so
those crazy heap spikes, your caches are essentially overflowing into
OldGen (with JNA).



On Tue, Nov 5, 2013 at 3:04 AM, Jiri Horky  wrote:

> Hi there,
>
> we are seeing extensive memory allocation leading to quite long and
> frequent GC pauses when using row cache. This is on cassandra 2.0.0
> cluster with JNA 4.0 library with following settings:
>
> key_cache_size_in_mb: 300
> key_cache_save_period: 14400
> row_cache_size_in_mb: 1024
> row_cache_save_period: 14400
> commitlog_sync: periodic
> commitlog_sync_period_in_ms: 1
> commitlog_segment_size_in_mb: 32
>
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10G -Xmx10G
> -Xmn1024M -XX:+HeapDumpOnOutOfMemoryError
>
> -XX:HeapDumpPath=/data2/cassandra-work/instance-1/cassandra-1383566283-pid1893.hprof
> -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+UseCondCardMark
>
> We have disabled row cache on one node to see  the  difference. Please
> see attached plots from visual VM, I think that the effect is quite
> visible. I have also taken 10x "jmap -histo" after 5s on a affected
> server and plotted the result, attached as well.
>
> I have taken a dump of the application when the heap size was 10GB, most
> of the memory was unreachable, which was expected. The majority was used
> by 55-59M objects of HeapByteBuffer, byte[] and
> org.apache.cassandra.db.Column classes. I also include a list of inbound
> references to the HeapByteBuffer objects from which it should be visible
> where they are being allocated. This was acquired using Eclipse MAT.
>
> Here is the comparison of GC times when row cache enabled and disabled:
>
> prg01 - row cache enabled
>   - uptime 20h45m
>   - ConcurrentMarkSweep - 11494686ms
>   - ParNew - 14690885 ms
>   - time spent in GC: 35%
> prg02 - row cache disabled
>   - uptime 23h45m
>   - ConcurrentMarkSweep - 251ms
>   - ParNew - 230791 ms
>   - time spent in GC: 0.27%
>
> I would be grateful for any hints. Please let me know if you need any
> further information. For now, we are going to disable the row cache.
>
> Regards
> Jiri Horky
>

Re: Multi-dc restart impact

[J. Ryan Earl]

Are you doing QUORUM reads instead of LOCAL_QUORUM reads?


On Wed, Oct 9, 2013 at 7:41 PM, Chris Burroughs
wrote:

> I have not been able to do the test with the 2nd cluster, but have been
> given a disturbing data point.  We had a disk slowly fail causing a
> significant performance degradation that was only resolved when the "sick"
> node was killed.
>  * Perf in DC w/ sick disk: 
> http://i.imgur.com/W1I5ymL.**png?1
>  * perf in other DC: 
> http://i.imgur.com/gEMrLyF.**png?1
>
> Not only was a single slow node able to cause an order of magnitude
> performance hit in a dc, but the other dc faired *worse*.
>
>
> On 09/18/2013 08:50 AM, Chris Burroughs wrote:
>
>> On 09/17/2013 04:44 PM, Robert Coli wrote:
>>
>>> On Thu, Sep 5, 2013 at 6:14 AM, Chris Burroughs
>>> **wrote:
>>>
>>>  We have a 2 DC cluster running cassandra 1.2.9.  They are in actual
 physically separate DCs on opposite coasts of the US, not just logical
 ones.  The primary use of this cluster is CL.ONE reads out of a single
 column family.  My expectation was that in such a scenario restarts
 would
 have minimal impact in the DC where the restart occurred, and no
 impact in
 the remote DC.

 We are seeing instead that restarts in one DC have a dramatic impact on
 performance in the other (let's call them DCs "A" and "B").


>>> Did you end up filing a JIRA on this, or some other outcome?
>>>
>>> =Rob
>>>
>>>
>>
>> No.  I am currently in the process of taking a 2nd cluster from being
>> single to dual DC.  Once that is done I was going to repeat the test
>> with each cluster and gather as much information as reasonable.
>>
>
>