[jira] [Commented] (CASSANDRA-10714) tcp retransmission issue seen in cassandra cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269110#comment-15269110 ] Jeff Liu commented on CASSANDRA-10714: -- [~JoshuaMcKenzie], Josh, I think it's related to the aws network. I'm seeing it on other hosts as well. > tcp retransmission issue seen in cassandra cluster > -- > > Key: CASSANDRA-10714 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10714 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Jeff Liu > > I have been seen tcp package retransmission issue in various stacks in our > environment. ( AWS with no VPC). I'm currently using hsha rpc server type on > cassandra 2.1.6 version. The information captured by wireshark shows that the > retransmission happened both between client-to-server and server-to-server. > Even within a cluster that doesn't have any client traffic, sporadic > retransmission still happens. > It's pretty easy to reproduce this issue by watching "netstat -s | grep > retrans". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9625) GraphiteReporter not reporting
[ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157368#comment-15157368 ] Jeff Liu commented on CASSANDRA-9625: - I'm also running a test for 2.1.13. Will keep you all posted. > GraphiteReporter not reporting > -- > > Key: CASSANDRA-9625 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9625 > Project: Cassandra > Issue Type: Bug > Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3 >Reporter: Eric Evans >Assignee: T Jake Luciani > Attachments: metrics.yaml, thread-dump.log > > > When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops > working. The usual startup is logged, and one batch of samples is sent, but > the reporting interval comes and goes, and no other samples are ever sent. > The logs are free from errors. > Frustratingly, metrics reporting works in our smaller (staging) environment > on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not > on a 3 node (otherwise identical) staging cluster (maybe it takes a certain > level of concurrency?). > Attached is a thread dump, and our metrics.yaml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9625) GraphiteReporter not reporting
[ https://issues.apache.org/jira/browse/CASSANDRA-9625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139488#comment-15139488 ] Jeff Liu commented on CASSANDRA-9625: - anyone has update on this ticket ? is it resolved on any later version? > GraphiteReporter not reporting > -- > > Key: CASSANDRA-9625 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9625 > Project: Cassandra > Issue Type: Bug > Environment: Debian Jessie, 7u79-2.5.5-1~deb8u1, Cassandra 2.1.3 >Reporter: Eric Evans > Attachments: metrics.yaml, thread-dump.log > > > When upgrading from 2.1.3 to 2.1.6, the Graphite metrics reporter stops > working. The usual startup is logged, and one batch of samples is sent, but > the reporting interval comes and goes, and no other samples are ever sent. > The logs are free from errors. > Frustratingly, metrics reporting works in our smaller (staging) environment > on 2.1.6; We are able to reproduce this on all 6 of production nodes, but not > on a 3 node (otherwise identical) staging cluster (maybe it takes a certain > level of concurrency?). > Attached is a thread dump, and our metrics.yaml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes
[ https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139291#comment-15139291 ] Jeff Liu commented on CASSANDRA-9945: - [~jasobrown], should I expect it to be ready in 3.4 version? > Add transparent data encryption core classes > > > Key: CASSANDRA-9945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9945 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown > Labels: encryption > Fix For: 3.2 > > > This patch will add the core infrastructure classes necessary for transparent > data encryption (file-level encryption), as required for CASSANDRA-6018 and > CASSANDRA-9633. The phrase "transparent data encryption", while not the most > aesthetically pleasing, seems to be used throughout the database industry > (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, > so we'll go with that, as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes
[ https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15137372#comment-15137372 ] Jeff Liu commented on CASSANDRA-9945: - Thanks, [~jjordan], do you know when should I expect the transparent encryption to be fully available? we have some business requirement that comes in later this year that requires encryption for data at rest. I'm trying to align up the schedule. > Add transparent data encryption core classes > > > Key: CASSANDRA-9945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9945 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown > Labels: encryption > Fix For: 3.2 > > > This patch will add the core infrastructure classes necessary for transparent > data encryption (file-level encryption), as required for CASSANDRA-6018 and > CASSANDRA-9633. The phrase "transparent data encryption", while not the most > aesthetically pleasing, seems to be used throughout the database industry > (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, > so we'll go with that, as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9945) Add transparent data encryption core classes
[ https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135221#comment-15135221 ] Jeff Liu edited comment on CASSANDRA-9945 at 2/5/16 11:09 PM: -- [~jasobrown] With this ticket being resolved, does it mean we can do transparent data encryption in 3.2 as what we can do with datastax enterprise version? More specifically, can I run the following command to encrypt data? ALTER TABLE dept WITH compression_parameters:sstable_compression = 'Encryptor' AND compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' AND compression_parameters:secret_key_strength = 128; Thanks, Jeff was (Author: jeffl): [~jasobrown] With this ticket resolved, does it mean we can do transparent data encryption in 3.2 as what we can do with datastax enterprise version? More specifically, can I run the following command to encrypt data? ALTER TABLE dept WITH compression_parameters:sstable_compression = 'Encryptor' AND compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' AND compression_parameters:secret_key_strength = 128; Thanks, Jeff > Add transparent data encryption core classes > > > Key: CASSANDRA-9945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9945 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown > Labels: encryption > Fix For: 3.2 > > > This patch will add the core infrastructure classes necessary for transparent > data encryption (file-level encryption), as required for CASSANDRA-6018 and > CASSANDRA-9633. The phrase "transparent data encryption", while not the most > aesthetically pleasing, seems to be used throughout the database industry > (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, > so we'll go with that, as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9945) Add transparent data encryption core classes
[ https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135221#comment-15135221 ] Jeff Liu edited comment on CASSANDRA-9945 at 2/5/16 11:09 PM: -- [~jasobrown] With this ticket resolved, does it mean we can do transparent data encryption in 3.2 as what we can do with datastax enterprise version? More specifically, can I run the following command to encrypt data? ALTER TABLE dept WITH compression_parameters:sstable_compression = 'Encryptor' AND compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' AND compression_parameters:secret_key_strength = 128; Thanks, Jeff was (Author: jeffl): With this ticket resolved, does it mean we can do transparent data encryption in 3.2 as what we can do with datastax enterprise version? More specifically, can I run the following command to encrypt data? ALTER TABLE dept WITH compression_parameters:sstable_compression = 'Encryptor' AND compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' AND compression_parameters:secret_key_strength = 128; Thanks, Jeff > Add transparent data encryption core classes > > > Key: CASSANDRA-9945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9945 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown > Labels: encryption > Fix For: 3.2 > > > This patch will add the core infrastructure classes necessary for transparent > data encryption (file-level encryption), as required for CASSANDRA-6018 and > CASSANDRA-9633. The phrase "transparent data encryption", while not the most > aesthetically pleasing, seems to be used throughout the database industry > (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, > so we'll go with that, as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes
[ https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135221#comment-15135221 ] Jeff Liu commented on CASSANDRA-9945: - With this ticket resolved, does it mean we can do transparent data encryption in 3.2 as what we can do with datastax enterprise version? More specifically, can I run the following command to encrypt data? ALTER TABLE dept WITH compression_parameters:sstable_compression = 'Encryptor' AND compression_parameters:cipher_algorithm = 'AES/ECB/PKCS5Padding' AND compression_parameters:secret_key_strength = 128; Thanks, Jeff > Add transparent data encryption core classes > > > Key: CASSANDRA-9945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9945 > Project: Cassandra > Issue Type: Improvement >Reporter: Jason Brown >Assignee: Jason Brown > Labels: encryption > Fix For: 3.2 > > > This patch will add the core infrastructure classes necessary for transparent > data encryption (file-level encryption), as required for CASSANDRA-6018 and > CASSANDRA-9633. The phrase "transparent data encryption", while not the most > aesthetically pleasing, seems to be used throughout the database industry > (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, > so we'll go with that, as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10714) tcp retransmission issue seen in cassandra cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-10714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009217#comment-15009217 ] Jeff Liu commented on CASSANDRA-10714: -- [~mshuler], I checked the NIC status and it shows errors:0 dropped:0 overruns:0 frame:0 on both rx and tx. I even increased the tx queue size to 5000 from default 1000 without any improvement. Specifically for linux networking settings, I also increased both net.core.rmem, net.core.wmem,, net.ipv4.tcp_rmem, net.ipv4.tcp_wmem to 32M. increasd net.core.netdev_max_backlog to 3 with no luck to eliminate the retransmission. > tcp retransmission issue seen in cassandra cluster > -- > > Key: CASSANDRA-10714 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10714 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Jeff Liu > > I have been seen tcp package retransmission issue in various stacks in our > environment. ( AWS with no VPC). I'm currently using hsha rpc server type on > cassandra 2.1.6 version. The information captured by wireshark shows that the > retransmission happened both between client-to-server and server-to-server. > Even within a cluster that doesn't have any client traffic, sporadic > retransmission still happens. > It's pretty easy to reproduce this issue by watching "netstat -s | grep > retrans". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10714) tcp retransmission issue seen in cassandra cluster
Jeff Liu created CASSANDRA-10714: Summary: tcp retransmission issue seen in cassandra cluster Key: CASSANDRA-10714 URL: https://issues.apache.org/jira/browse/CASSANDRA-10714 Project: Cassandra Issue Type: Improvement Components: Streaming and Messaging Reporter: Jeff Liu I have been seen tcp package retransmission issue in various stacks in our environment. ( AWS with no VPC). I'm currently using hsha rpc server type on cassandra 2.1.6 version. The information captured by wireshark shows that the retransmission happened both between client-to-server and server-to-server. Even within a cluster that doesn't have any client traffic, sporadic retransmission still happens. It's pretty easy to reproduce this issue by watching "netstat -s | grep retrans". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10713) Gate keeper to do rate limiter and qps cap
[ https://issues.apache.org/jira/browse/CASSANDRA-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Liu updated CASSANDRA-10713: - Description: As cassandra becomes the primary noSQL data store in more and more production environment, have we thought about adding gate keeper features like rate limiter and qps cap, which will give a greater integration experience when implementing cassandra in a large services infrastructure. Reliability has become more and more important in those days for service providers. cassandra, together with other SQL or noSQL solutions, provides the data layers that power the complete application stack. In today's distributed system frameworks, dependences across application layers have been largely reduced, however, as the central data repository, data store system sits on the critical path, and become more fragile especially for the real-time, large scale systems. Companies like Twitter, facebook has built gate keeper internally to protect their data store systems, what should we do for cassandra? Thoughts? was: As cassandra becomes the primary noSQL data store in more and more production environment, have we thought about adding gate keeper features like rate limiter and qps cap, which will give a greater integration experience when implementing cassandra in a large services infrastructure. Thoughts? > Gate keeper to do rate limiter and qps cap > -- > > Key: CASSANDRA-10713 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10713 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeff Liu > > As cassandra becomes the primary noSQL data store in more and more production > environment, have we thought about adding gate keeper features like rate > limiter and qps cap, which will give a greater integration experience when > implementing cassandra in a large services infrastructure. > Reliability has become more and more important in those days for service > providers. cassandra, together with other SQL or noSQL solutions, provides > the data layers that power the complete application stack. In today's > distributed system frameworks, dependences across application layers have > been largely reduced, however, as the central data repository, data store > system sits on the critical path, and become more fragile especially for the > real-time, large scale systems. Companies like Twitter, facebook has built > gate keeper internally to protect their data store systems, what should we do > for cassandra? > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10713) Gate keeper to do rate limiter and qps cap
Jeff Liu created CASSANDRA-10713: Summary: Gate keeper to do rate limiter and qps cap Key: CASSANDRA-10713 URL: https://issues.apache.org/jira/browse/CASSANDRA-10713 Project: Cassandra Issue Type: Improvement Components: Configuration Reporter: Jeff Liu As cassandra becomes the primary noSQL data store in more and more production environment, have we thought about adding gate keeper features like rate limiter and qps cap, which will give a greater integration experience when implementing cassandra in a large services infrastructure. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9804) Periodically Client Timeout Seen in Cassandra 1.2.9
Jeff Liu created CASSANDRA-9804:
---
Summary: Periodically Client Timeout Seen in Cassandra 1.2.9
Key: CASSANDRA-9804
URL: https://issues.apache.org/jira/browse/CASSANDRA-9804
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Jeff Liu
We are seeing some Timout exception from various clients that connect to
cassandra in our environment. It happens very randomly at a frequency about one
very short 2 mini-seconds in around 36 hours.
The TimeoutException from client side looks like the following:
{noformat}
WARNING impl.Slf4jConnectionPoolMonitorImpl [tid-394]
[request_id=a8f48aa0-2a35-11e5-bce6-22000b4b0a4b] : OperationTimeoutException:
[host=10.136.6.98(10.136.6.98):9160, latency=10001(10001),
attempts=1]TimedOutException()
{noformat}
Checked system.log on server side and no log entries at all the time duration
when the client timeout happened.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530863#comment-14530863
]
Jeff Liu commented on CASSANDRA-8723:
-
Just returned from vacation.
We have been running with 2.0.12 quite stable while waiting for 2.1.3. I'll
schedule an upgrade sometime soon.
Thanks,
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.x
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8899) cqlsh - not able to get row count with select(*) for large table
[
https://issues.apache.org/jira/browse/CASSANDRA-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503876#comment-14503876
]
Jeff Liu commented on CASSANDRA-8899:
-
Yes, [~blerer]. The problem is worked around by increasing timeout value.
> cqlsh - not able to get row count with select(*) for large table
>
>
> Key: CASSANDRA-8899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8899
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.2 ubuntu12.04
>Reporter: Jeff Liu
>Assignee: Benjamin Lerer
>
> I'm getting errors when running a query that looks at a large number of rows.
> {noformat}
> cqlsh:events> select count(*) from catalog;
> count
> ---
> 1
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 11000;
> count
> ---
> 11000
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 5;
> errors={}, last_host=127.0.0.1
> cqlsh:events>
> {noformat}
> We are not able to make the select * query to get row count.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367499#comment-14367499
]
Jeff Liu commented on CASSANDRA-8696:
-
Unfortunately I had to rebuild the cluster and keyspace and I don't have an
environment handy to debug this issue. [~ranrub] you asked about the syntax to
enable DEBUG in logback.xml. Do you still have the environment ready to debug?
Thanks!
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.repair.RepairJob$2.onFailure(R
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367470#comment-14367470
]
Jeff Liu commented on CASSANDRA-8696:
-
[~benedict] and team, there have been several people reporting the same issue
since I created the ticket. Any progress on reproducing and identifying the
root cause? Thanks.
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
[jira] [Resolved] (CASSANDRA-8968) Cassandra cqlsh query return different result randomly
[
https://issues.apache.org/jira/browse/CASSANDRA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu resolved CASSANDRA-8968.
-
Resolution: Fixed
> Cassandra cqlsh query return different result randomly
> --
>
> Key: CASSANDRA-8968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8968
> Project: Cassandra
> Issue Type: Wish
> Environment: Cassandra 2.0
>Reporter: Jeff Liu
>
> Noticed that a select query in cqlsh returns different result randomly. It
> would be nice to get consistent result when same query is performed.
> {noformat}
> cqlsh> select * from cass_dc.cass_dc ;
> key | value
> -+---
>c | ccc
>b | bbb
> Tracing session: d20490f0-c9ac-11e4-a527-23b8e6fcc12b
> activity| timestamp|
> source | source_elapsed
> -+--+--+
> execute_cql3_query | 18:14:42,305 |
> 54.92.168.12 | 0
> Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:42,306 |
> 54.92.168.12 |855
> Preparing statement | 18:14:42,307 |
> 54.92.168.12 | 1950
>Determining replicas to query | 18:14:42,307 |
> 54.92.168.12 | 2101
> Message received from /x.x.x.12 | 18:14:42,308 |
> 54.82.42.121 | 56
> Enqueuing request to /x.x.x.121 | 18:14:42,308 |
> 54.92.168.12 | 2685
> Sending message to /x.x.x.121 | 18:14:42,308 |
> 54.92.168.12 | 2825
> Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:42,309 |
> 54.82.42.121 |556
> Read 1 live and 0 tombstoned cells | 18:14:42,309 |
> 54.82.42.121 |868
> Read 1 live and 0 tombstoned cells | 18:14:42,309 |
> 54.82.42.121 |956
> Scanned 2 rows and matched 2 | 18:14:42,309 |
> 54.82.42.121 |989
> Enqueuing response to /x.x.x.12 | 18:14:42,309 |
> 54.82.42.121 | 1007
> Sending message to /x.x.x.12 | 18:14:42,310 |
> 54.82.42.121 | 1287
> Message received from /x.x.x.121 | 18:14:42,319 |
> 54.92.168.12 | 13656
> Processing response from /x.x.x.121 | 18:14:42,319 |
> 54.92.168.12 | 14133
> Request complete | 18:14:42,319 |
> 54.92.168.12 | 14808
> cqlsh> select * from cass_dc.cass_dc ;
> key | value
> -+---
>a | aaa
>c | ccc
>b | bbb
> Tracing session: d4ecbcc0-c9ac-11e4-a527-23b8e6fcc12b
> activity| timestamp|
> source | source_elapsed
> -+--+--+
> execute_cql3_query | 18:14:47,180 |
> 54.92.168.12 | 0
> Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:47,180 |
> 54.92.168.12 | 81
> Preparing statement | 18:14:47,181 |
> 54.92.168.12 |224
>Determining replicas to query | 18:14:47,181 |
> 54.92.168.12 |383
> Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:47,184 |
> 54.92.168.12 | 3611
> Read 1 live and 0 tombstoned cells | 18:14:47,184 |
> 54.92.168.12 | 4073
> Read 1 live and 0 tombstoned cells | 18:14:47,185 |
> 54.92.168.12 | 4239
> Read 1 live and 0 tombstoned cells | 18:14:47,185 |
> 54.92.168.12 | 4559
> Scanned 3 rows and matched 3 | 18:14:47,185 |
> 54.92.168.12 | 4601
> Request complete | 18:14:47,185 |
> 54.92.168.12 | 5812
> cqlsh> select * from cass_dc.cass_dc ;
> key | value
> -+---
>a | aaa
>c | ccc
>b | bbb
> Tracing session: 10247f30-c9ad-11e4-a527-23b8e6fcc12b
> activity| timestamp|
> source | source_elapsed
> -+--+--+
> execute_cql3_query | 18:16:26,531 |
> 54.92.168.12 | 0
> Parsing select * from cass_dc.cass_dc
[jira] [Commented] (CASSANDRA-8968) Cassandra cqlsh query return different result randomly
[
https://issues.apache.org/jira/browse/CASSANDRA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360948#comment-14360948
]
Jeff Liu commented on CASSANDRA-8968:
-
That works. Thanks.
> Cassandra cqlsh query return different result randomly
> --
>
> Key: CASSANDRA-8968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8968
> Project: Cassandra
> Issue Type: Wish
> Environment: Cassandra 2.0
>Reporter: Jeff Liu
>
> Noticed that a select query in cqlsh returns different result randomly. It
> would be nice to get consistent result when same query is performed.
> {noformat}
> cqlsh> select * from cass_dc.cass_dc ;
> key | value
> -+---
>c | ccc
>b | bbb
> Tracing session: d20490f0-c9ac-11e4-a527-23b8e6fcc12b
> activity| timestamp|
> source | source_elapsed
> -+--+--+
> execute_cql3_query | 18:14:42,305 |
> 54.92.168.12 | 0
> Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:42,306 |
> 54.92.168.12 |855
> Preparing statement | 18:14:42,307 |
> 54.92.168.12 | 1950
>Determining replicas to query | 18:14:42,307 |
> 54.92.168.12 | 2101
> Message received from /x.x.x.12 | 18:14:42,308 |
> 54.82.42.121 | 56
> Enqueuing request to /x.x.x.121 | 18:14:42,308 |
> 54.92.168.12 | 2685
> Sending message to /x.x.x.121 | 18:14:42,308 |
> 54.92.168.12 | 2825
> Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:42,309 |
> 54.82.42.121 |556
> Read 1 live and 0 tombstoned cells | 18:14:42,309 |
> 54.82.42.121 |868
> Read 1 live and 0 tombstoned cells | 18:14:42,309 |
> 54.82.42.121 |956
> Scanned 2 rows and matched 2 | 18:14:42,309 |
> 54.82.42.121 |989
> Enqueuing response to /x.x.x.12 | 18:14:42,309 |
> 54.82.42.121 | 1007
> Sending message to /x.x.x.12 | 18:14:42,310 |
> 54.82.42.121 | 1287
> Message received from /x.x.x.121 | 18:14:42,319 |
> 54.92.168.12 | 13656
> Processing response from /x.x.x.121 | 18:14:42,319 |
> 54.92.168.12 | 14133
> Request complete | 18:14:42,319 |
> 54.92.168.12 | 14808
> cqlsh> select * from cass_dc.cass_dc ;
> key | value
> -+---
>a | aaa
>c | ccc
>b | bbb
> Tracing session: d4ecbcc0-c9ac-11e4-a527-23b8e6fcc12b
> activity| timestamp|
> source | source_elapsed
> -+--+--+
> execute_cql3_query | 18:14:47,180 |
> 54.92.168.12 | 0
> Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:47,180 |
> 54.92.168.12 | 81
> Preparing statement | 18:14:47,181 |
> 54.92.168.12 |224
>Determining replicas to query | 18:14:47,181 |
> 54.92.168.12 |383
> Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:47,184 |
> 54.92.168.12 | 3611
> Read 1 live and 0 tombstoned cells | 18:14:47,184 |
> 54.92.168.12 | 4073
> Read 1 live and 0 tombstoned cells | 18:14:47,185 |
> 54.92.168.12 | 4239
> Read 1 live and 0 tombstoned cells | 18:14:47,185 |
> 54.92.168.12 | 4559
> Scanned 3 rows and matched 3 | 18:14:47,185 |
> 54.92.168.12 | 4601
> Request complete | 18:14:47,185 |
> 54.92.168.12 | 5812
> cqlsh> select * from cass_dc.cass_dc ;
> key | value
> -+---
>a | aaa
>c | ccc
>b | bbb
> Tracing session: 10247f30-c9ad-11e4-a527-23b8e6fcc12b
> activity| timestamp|
> source | source_elapsed
> -+--+--+
> execute_cql3_query | 18:16:26,531 |
> 54.92.168.12 |
[jira] [Updated] (CASSANDRA-8968) Cassandra cqlsh query return different result randomly
[
https://issues.apache.org/jira/browse/CASSANDRA-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8968:
Description:
Noticed that a select query in cqlsh returns different result randomly. It
would be nice to get consistent result when same query is performed.
{noformat}
cqlsh> select * from cass_dc.cass_dc ;
key | value
-+---
c | ccc
b | bbb
Tracing session: d20490f0-c9ac-11e4-a527-23b8e6fcc12b
activity| timestamp|
source | source_elapsed
-+--+--+
execute_cql3_query | 18:14:42,305 |
54.92.168.12 | 0
Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:42,306 |
54.92.168.12 |855
Preparing statement | 18:14:42,307 |
54.92.168.12 | 1950
Determining replicas to query | 18:14:42,307 |
54.92.168.12 | 2101
Message received from /x.x.x.12 | 18:14:42,308 |
54.82.42.121 | 56
Enqueuing request to /x.x.x.121 | 18:14:42,308 |
54.92.168.12 | 2685
Sending message to /x.x.x.121 | 18:14:42,308 |
54.92.168.12 | 2825
Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:42,309 |
54.82.42.121 |556
Read 1 live and 0 tombstoned cells | 18:14:42,309 |
54.82.42.121 |868
Read 1 live and 0 tombstoned cells | 18:14:42,309 |
54.82.42.121 |956
Scanned 2 rows and matched 2 | 18:14:42,309 |
54.82.42.121 |989
Enqueuing response to /x.x.x.12 | 18:14:42,309 |
54.82.42.121 | 1007
Sending message to /x.x.x.12 | 18:14:42,310 |
54.82.42.121 | 1287
Message received from /x.x.x.121 | 18:14:42,319 |
54.92.168.12 | 13656
Processing response from /x.x.x.121 | 18:14:42,319 |
54.92.168.12 | 14133
Request complete | 18:14:42,319 |
54.92.168.12 | 14808
cqlsh> select * from cass_dc.cass_dc ;
key | value
-+---
a | aaa
c | ccc
b | bbb
Tracing session: d4ecbcc0-c9ac-11e4-a527-23b8e6fcc12b
activity| timestamp|
source | source_elapsed
-+--+--+
execute_cql3_query | 18:14:47,180 |
54.92.168.12 | 0
Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:47,180 |
54.92.168.12 | 81
Preparing statement | 18:14:47,181 |
54.92.168.12 |224
Determining replicas to query | 18:14:47,181 |
54.92.168.12 |383
Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:47,184 |
54.92.168.12 | 3611
Read 1 live and 0 tombstoned cells | 18:14:47,184 |
54.92.168.12 | 4073
Read 1 live and 0 tombstoned cells | 18:14:47,185 |
54.92.168.12 | 4239
Read 1 live and 0 tombstoned cells | 18:14:47,185 |
54.92.168.12 | 4559
Scanned 3 rows and matched 3 | 18:14:47,185 |
54.92.168.12 | 4601
Request complete | 18:14:47,185 |
54.92.168.12 | 5812
cqlsh> select * from cass_dc.cass_dc ;
key | value
-+---
a | aaa
c | ccc
b | bbb
Tracing session: 10247f30-c9ad-11e4-a527-23b8e6fcc12b
activity| timestamp|
source | source_elapsed
-+--+--+
execute_cql3_query | 18:16:26,531 |
54.92.168.12 | 0
Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:16:26,531 |
54.92.168.12 |116
Preparing statement | 18:16:26,531 |
54.92.168.12 |237
Determining replicas to query | 18:16:26,531 |
54.92.168.12 |386
Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:16:26,532 |
54.92.168.12 | 1280
Read 1 live and 0 tombstoned cells | 18:16:26,533 |
54.92.168.12 | 1447
Rea
[jira] [Created] (CASSANDRA-8968) Cassandra cqlsh query return different result randomly
Jeff Liu created CASSANDRA-8968:
---
Summary: Cassandra cqlsh query return different result randomly
Key: CASSANDRA-8968
URL: https://issues.apache.org/jira/browse/CASSANDRA-8968
Project: Cassandra
Issue Type: Wish
Environment: Cassandra 2.0
Reporter: Jeff Liu
Noticed that a select query in cqlsh returns different result randomly. It
would be nice to get consistent result when same query is performed.
{noformat}
cqlsh> select * from cass_dc.cass_dc ;
key | value
-+---
c | ccc
b | bbb
Tracing session: d20490f0-c9ac-11e4-a527-23b8e6fcc12b
activity| timestamp|
source | source_elapsed
-+--+--+
execute_cql3_query | 18:14:42,305 |
54.92.168.12 | 0
Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:42,306 |
54.92.168.12 |855
Preparing statement | 18:14:42,307 |
54.92.168.12 | 1950
Determining replicas to query | 18:14:42,307 |
54.92.168.12 | 2101
Message received from /54.92.168.12 | 18:14:42,308 |
54.82.42.121 | 56
Enqueuing request to /54.82.42.121 | 18:14:42,308 |
54.92.168.12 | 2685
Sending message to /54.82.42.121 | 18:14:42,308 |
54.92.168.12 | 2825
Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:42,309 |
54.82.42.121 |556
Read 1 live and 0 tombstoned cells | 18:14:42,309 |
54.82.42.121 |868
Read 1 live and 0 tombstoned cells | 18:14:42,309 |
54.82.42.121 |956
Scanned 2 rows and matched 2 | 18:14:42,309 |
54.82.42.121 |989
Enqueuing response to /54.92.168.12 | 18:14:42,309 |
54.82.42.121 | 1007
Sending message to /54.92.168.12 | 18:14:42,310 |
54.82.42.121 | 1287
Message received from /54.82.42.121 | 18:14:42,319 |
54.92.168.12 | 13656
Processing response from /54.82.42.121 | 18:14:42,319 |
54.92.168.12 | 14133
Request complete | 18:14:42,319 |
54.92.168.12 | 14808
cqlsh> select * from cass_dc.cass_dc ;
key | value
-+---
a | aaa
c | ccc
b | bbb
Tracing session: d4ecbcc0-c9ac-11e4-a527-23b8e6fcc12b
activity| timestamp|
source | source_elapsed
-+--+--+
execute_cql3_query | 18:14:47,180 |
54.92.168.12 | 0
Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:14:47,180 |
54.92.168.12 | 81
Preparing statement | 18:14:47,181 |
54.92.168.12 |224
Determining replicas to query | 18:14:47,181 |
54.92.168.12 |383
Executing seq scan across 0 sstables for [min(-1), min(-1)] | 18:14:47,184 |
54.92.168.12 | 3611
Read 1 live and 0 tombstoned cells | 18:14:47,184 |
54.92.168.12 | 4073
Read 1 live and 0 tombstoned cells | 18:14:47,185 |
54.92.168.12 | 4239
Read 1 live and 0 tombstoned cells | 18:14:47,185 |
54.92.168.12 | 4559
Scanned 3 rows and matched 3 | 18:14:47,185 |
54.92.168.12 | 4601
Request complete | 18:14:47,185 |
54.92.168.12 | 5812
cqlsh> select * from cass_dc.cass_dc ;
key | value
-+---
a | aaa
c | ccc
b | bbb
Tracing session: 10247f30-c9ad-11e4-a527-23b8e6fcc12b
activity| timestamp|
source | source_elapsed
-+--+--+
execute_cql3_query | 18:16:26,531 |
54.92.168.12 | 0
Parsing select * from cass_dc.cass_dc LIMIT 1; | 18:16:26,531 |
54.92.168.12 |116
Preparing statement | 18:16:26,531 |
54.92.168.12 |237
Determining replicas to query | 18:16:26,531 |
54.92.168.12 |386
Executing seq scan across 0 sstables for [min(-1),
[jira] [Created] (CASSANDRA-8899) cqlsh not able to get row count with select(*) with large table
Jeff Liu created CASSANDRA-8899:
---
Summary: cqlsh not able to get row count with select(*) with large
table
Key: CASSANDRA-8899
URL: https://issues.apache.org/jira/browse/CASSANDRA-8899
Project: Cassandra
Issue Type: Bug
Environment: Cassandra 2.1.2 ubuntu12.04
Reporter: Jeff Liu
I'm getting errors when running a query that looks at a large number of rows.
{noformat}
cqlsh:events> select count(*) from catalog;
count
---
1
(1 rows)
cqlsh:events> select count(*) from catalog limit 11000;
count
---
11000
(1 rows)
cqlsh:events> select count(*) from catalog limit 5;
errors={}, last_host=127.0.0.1
cqlsh:events>
{noformat}
We don't make queries w/o a WHERE clause in Chisel itself but I can't validate
the correct number of rows are being inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8899) cqlsh - not able to get row count with select(*) for large table
[
https://issues.apache.org/jira/browse/CASSANDRA-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8899:
Description:
I'm getting errors when running a query that looks at a large number of rows.
{noformat}
cqlsh:events> select count(*) from catalog;
count
---
1
(1 rows)
cqlsh:events> select count(*) from catalog limit 11000;
count
---
11000
(1 rows)
cqlsh:events> select count(*) from catalog limit 5;
errors={}, last_host=127.0.0.1
cqlsh:events>
{noformat}
We are not able to make the select * query to get row count.
was:
I'm getting errors when running a query that looks at a large number of rows.
{noformat}
cqlsh:events> select count(*) from catalog;
count
---
1
(1 rows)
cqlsh:events> select count(*) from catalog limit 11000;
count
---
11000
(1 rows)
cqlsh:events> select count(*) from catalog limit 5;
errors={}, last_host=127.0.0.1
cqlsh:events>
{noformat}
We are not able to make the select(*) query to get row count.
> cqlsh - not able to get row count with select(*) for large table
>
>
> Key: CASSANDRA-8899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8899
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.2 ubuntu12.04
>Reporter: Jeff Liu
>
> I'm getting errors when running a query that looks at a large number of rows.
> {noformat}
> cqlsh:events> select count(*) from catalog;
> count
> ---
> 1
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 11000;
> count
> ---
> 11000
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 5;
> errors={}, last_host=127.0.0.1
> cqlsh:events>
> {noformat}
> We are not able to make the select * query to get row count.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8899) cqlsh - not able to get row count with select(*) for large table
[
https://issues.apache.org/jira/browse/CASSANDRA-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8899:
Description:
I'm getting errors when running a query that looks at a large number of rows.
{noformat}
cqlsh:events> select count(*) from catalog;
count
---
1
(1 rows)
cqlsh:events> select count(*) from catalog limit 11000;
count
---
11000
(1 rows)
cqlsh:events> select count(*) from catalog limit 5;
errors={}, last_host=127.0.0.1
cqlsh:events>
{noformat}
We are not able to make the select(*) query to get row count.
was:
I'm getting errors when running a query that looks at a large number of rows.
{noformat}
cqlsh:events> select count(*) from catalog;
count
---
1
(1 rows)
cqlsh:events> select count(*) from catalog limit 11000;
count
---
11000
(1 rows)
cqlsh:events> select count(*) from catalog limit 5;
errors={}, last_host=127.0.0.1
cqlsh:events>
{noformat}
We don't make queries w/o a WHERE clause in Chisel itself but I can't validate
the correct number of rows are being inserted into the table.
> cqlsh - not able to get row count with select(*) for large table
>
>
> Key: CASSANDRA-8899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8899
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.2 ubuntu12.04
>Reporter: Jeff Liu
>
> I'm getting errors when running a query that looks at a large number of rows.
> {noformat}
> cqlsh:events> select count(*) from catalog;
> count
> ---
> 1
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 11000;
> count
> ---
> 11000
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 5;
> errors={}, last_host=127.0.0.1
> cqlsh:events>
> {noformat}
> We are not able to make the select(*) query to get row count.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8899) cqlsh - not able to get row count with select(*) with large table
[
https://issues.apache.org/jira/browse/CASSANDRA-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8899:
Summary: cqlsh - not able to get row count with select(*) with large table
(was: cqlsh not able to get row count with select(*) with large table)
> cqlsh - not able to get row count with select(*) with large table
> -
>
> Key: CASSANDRA-8899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8899
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.2 ubuntu12.04
>Reporter: Jeff Liu
>
> I'm getting errors when running a query that looks at a large number of rows.
> {noformat}
> cqlsh:events> select count(*) from catalog;
> count
> ---
> 1
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 11000;
> count
> ---
> 11000
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 5;
> errors={}, last_host=127.0.0.1
> cqlsh:events>
> {noformat}
> We don't make queries w/o a WHERE clause in Chisel itself but I can't
> validate the correct number of rows are being inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8899) cqlsh - not able to get row count with select(*) for large table
[
https://issues.apache.org/jira/browse/CASSANDRA-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8899:
Summary: cqlsh - not able to get row count with select(*) for large table
(was: cqlsh - not able to get row count with select(*) with large table)
> cqlsh - not able to get row count with select(*) for large table
>
>
> Key: CASSANDRA-8899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8899
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.2 ubuntu12.04
>Reporter: Jeff Liu
>
> I'm getting errors when running a query that looks at a large number of rows.
> {noformat}
> cqlsh:events> select count(*) from catalog;
> count
> ---
> 1
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 11000;
> count
> ---
> 11000
> (1 rows)
> cqlsh:events> select count(*) from catalog limit 5;
> errors={}, last_host=127.0.0.1
> cqlsh:events>
> {noformat}
> We don't make queries w/o a WHERE clause in Chisel itself but I can't
> validate the correct number of rows are being inserted into the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8870) Tombstone overwhelming issue aborts client queries
[
https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345461#comment-14345461
]
Jeff Liu commented on CASSANDRA-8870:
-
Another question I have been curious about is that why we would see those
tombstone errors. in our application, we are doing insert and update only. Will
update ops generate tombstones?
> Tombstone overwhelming issue aborts client queries
> --
>
> Key: CASSANDRA-8870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8870
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra 2.1.2 ubunbtu 12.04
>Reporter: Jeff Liu
>
> We are getting client queries timeout issues on the clients who are trying to
> query data from cassandra cluster.
> Nodetool status shows that all nodes are still up regardless.
> Logs from client side:
> {noformat}
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried:
> cass-chisel01.abc01.abc02.abc.abc.com/10.66.182.113:9042
> (com.datastax.driver.core.TransportException:
> [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
> has been closed))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
> Logs from cassandra/system.log
> {noformat}
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
> Scanned over 10 tombstones in system.hints; query aborted (see
> tombstone_failure_threshold)
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 -
> Exception in thread Thread[HintedHandoff:2,1,main]
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8870) Tombstone overwhelming issue aborts client queries
[
https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8870:
Description:
We are getting client queries timeout issues on the clients who are trying to
query data from cassandra cluster.
Nodetool status shows that all nodes are still up regardless.
Logs from client side:
{noformat}
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried:
cass-chisel01.abc01.abc02.abc.abc.com/10.66.182.113:9042
(com.datastax.driver.core.TransportException:
[cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
has been closed))
at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
{noformat}
Logs from cassandra/system.log
{noformat}
ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
Scanned over 10 tombstones in system.hints; query aborted (see
tombstone_failure_threshold)
ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 -
Exception in thread Thread[HintedHandoff:2,1,main]
org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
{noformat}
was:
We are getting client queries timeout issues on the clients who are trying to
query data from cassandra cluster.
Nodetool status shows that all nodes are still up regardless.
Logs from client side:
{noformat}
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried:
cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042
(com.datastax.driver.core.TransportException:
[cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
has been closed))
at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
{noformat}
Logs from cassandra/system.log
{noformat}
ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
Scanned over 10 tombstones in system.hints; query aborted (see
tombst
[jira] [Comment Edited] (CASSANDRA-8870) Tombstone overwhelming issue aborts client queries
[
https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338890#comment-14338890
]
Jeff Liu edited comment on CASSANDRA-8870 at 2/26/15 6:49 PM:
--
Thanks Brandon. Just want to point it out that we are running into this issue
in version 2.1.2. CASSANDRA-6998 said to have been fixed at 2.1.1
was (Author: jeffl):
we are running into this issue in version 2.1.2. CASSANDRA-6998 said to have
been fixed at 2.1.1
> Tombstone overwhelming issue aborts client queries
> --
>
> Key: CASSANDRA-8870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8870
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra 2.1.2 ubunbtu 12.04
>Reporter: Jeff Liu
>
> We are getting client queries timeout issues on the clients who are trying to
> query data from cassandra cluster.
> Nodetool status shows that all nodes are still up regardless.
> Logs from client side:
> {noformat}
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried:
> cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042
> (com.datastax.driver.core.TransportException:
> [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
> has been closed))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
> Logs from cassandra/system.log
> {noformat}
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
> Scanned over 10 tombstones in system.hints; query aborted (see
> tombstone_failure_threshold)
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 -
> Exception in thread Thread[HintedHandoff:2,1,main]
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8870) Tombstone overwhelming issue aborts client queries
[
https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338890#comment-14338890
]
Jeff Liu commented on CASSANDRA-8870:
-
we are running into this issue in version 2.1.2. CASSANDRA-6998 said to have
been fixed at 2.1.1
> Tombstone overwhelming issue aborts client queries
> --
>
> Key: CASSANDRA-8870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8870
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra 2.1.2 ubunbtu 12.04
>Reporter: Jeff Liu
>
> We are getting client queries timeout issues on the clients who are trying to
> query data from cassandra cluster.
> Nodetool status shows that all nodes are still up regardless.
> Logs from client side:
> {noformat}
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried:
> cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042
> (com.datastax.driver.core.TransportException:
> [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
> has been closed))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
> Logs from cassandra/system.log
> {noformat}
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
> Scanned over 10 tombstones in system.hints; query aborted (see
> tombstone_failure_threshold)
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 -
> Exception in thread Thread[HintedHandoff:2,1,main]
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8870) Tombstone overwhelming issue aborts client queries
[
https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8870:
Summary: Tombstone overwhelming issue aborts client queries (was:
Tombstone overwhelming issue abort client queries)
> Tombstone overwhelming issue aborts client queries
> --
>
> Key: CASSANDRA-8870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8870
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra 2.1.2 ubunbtu 12.04
>Reporter: Jeff Liu
>
> We are getting client queries timeout issues on the clients who are trying to
> query data from cassandra cluster.
> Nodetool status shows that all nodes are still up regardless.
> Logs from client side:
> {noformat}
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried:
> cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042
> (com.datastax.driver.core.TransportException:
> [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
> has been closed))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
> Logs from cassandra/system.log
> {noformat}
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
> Scanned over 10 tombstones in system.hints; query aborted (see
> tombstone_failure_threshold)
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 -
> Exception in thread Thread[HintedHandoff:2,1,main]
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8870) Tombstone overwhelming issue abort client queries
[
https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8870:
Description:
We are getting client queries timeout issues on the clients who are trying to
query data from cassandra cluster.
Nodetool status shows that all nodes are still up regardless.
Logs from client side:
{noformat}
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried:
cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042
(com.datastax.driver.core.TransportException:
[cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
has been closed))
at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
{noformat}
Logs from cassandra/system.log
{noformat}
ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
Scanned over 10 tombstones in system.hints; query aborted (see
tombstone_failure_threshold)
ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 -
Exception in thread Thread[HintedHandoff:2,1,main]
org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
{noformat}
was:
We are getting client queries timeout issues on the clients who are trying to
query data from cassandra cluster.
Nodetool status shows that all nodes are still up regardless.
Logs from client side:
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried
for query failed (tried:
cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042
(com.datastax.driver.core.TransportException:
[cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
has been closed))
at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_55]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_55]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
Logs from cassandra/system.log
ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
Scanned over 10 tombstones in system.hints; query aborted (see
tombstone_failure_threshold)
[jira] [Updated] (CASSANDRA-8870) Tombstone overwhelming issue abort client queries
[ https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Liu updated CASSANDRA-8870: Description: We are getting client queries timeout issues on the clients who are trying to query data from cassandra cluster. Nodetool status shows that all nodes are still up regardless. Logs from client side: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042 (com.datastax.driver.core.TransportException: [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection has been closed)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na] at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55] Logs from cassandra/system.log ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 - Scanned over 10 tombstones in system.hints; query aborted (see tombstone_failure_threshold) ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 - Exception in thread Thread[HintedHandoff:2,1,main] org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555) ~[apache-cassandra-2.1.2.jar:2.1.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55] was: We are getting client queries timeout issues on the clients who are trying to query data from cassandra cluster. Nodetool status shows that all nodes are still up regardless. Logs from client side: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042 (com.datastax.driver.core.TransportException: [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection has been closed)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na] at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55] Logs from cassandra/system.log ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 - Scanned over 10 tombstones in system.hints; query aborted (see tombstone_failure_threshold) ERROR [HintedHandoff:2] 2015-02-23 23:46
[jira] [Created] (CASSANDRA-8870) Tombstone overwhelming issue abort client queries
Jeff Liu created CASSANDRA-8870: --- Summary: Tombstone overwhelming issue abort client queries Key: CASSANDRA-8870 URL: https://issues.apache.org/jira/browse/CASSANDRA-8870 Project: Cassandra Issue Type: Bug Environment: cassandra 2.1.2 ubunbtu 12.04 Reporter: Jeff Liu We are getting client queries timeout issues on the clients who are trying to query data from cassandra cluster. Nodetool status shows that all nodes are still up regardless. Logs from client side: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042 (com.datastax.driver.core.TransportException: [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection has been closed)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na] at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179) ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55] Logs from cassandra/system.log ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 - Scanned over 10 tombstones in system.hints; query aborted (see tombstone_failure_threshold) ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 - Exception in thread Thread[HintedHandoff:2,1,main] org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555) ~[apache-cassandra-2.1.2.jar:2.1.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14333686#comment-14333686
]
Jeff Liu commented on CASSANDRA-8723:
-
[~benedict] what is the release schedule for 2.1.3 and 2.1.4? if an estimate is
possible at this time?
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.4
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311779#comment-14311779
]
Jeff Liu commented on CASSANDRA-8723:
-
Thank you for your input [~thebrenthaines]. Glad that you were able to work
around it. How were you able to identify the problematic CF in the first place?
I haven't been able to find any issue, data related or not, in my cluster
before it went into OOM killing java situation. we are just steaming data into
the cluster at this stage. And because of this issue, we had to rebuild the
cluster couple times but still no luck to resolve the OOM killing issue.
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/ca
[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310075#comment-14310075
]
Jeff Liu commented on CASSANDRA-8723:
-
yes, we know c1.xlarge is relatively small in terms of the memory resource for
cassandra. We haven't identified any other resource contention on this instance
type with our application is the reason that we haven't turned to another high
capacity instance type. Also c1.xlarge gave us a cheap solution with enough
cheap storage.
The swap is enabled and swappiness is set at 60%.
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hp
[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309943#comment-14309943
]
Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:27 PM:
-
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
4. nodetool status show that the nodes are having over 40GB data/load:
{noformat}
-- Address Load Tokens OwnsHost ID
Rack
UN 10.98.xxx.5343.62 GB 256 ?
1ab1fec7-383b-4801-b899-059ea0c18def us-east-1d
UN 10.182.xxx.24 46.53 GB 256 ?
07a9849b-73f2-4166-a8e9-bedd40a1f30e us-east-1e
UN 10.32.xxx.222 62.36 GB 256 ?
fb8c38fc-0eb1-41c2-96c3-fc597aeede10 us-east-1d
UN 10.66.xxx.48 45.03 GB 256 ?
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2 us-east-1c
UN 10.232.xxx.154 50.57 GB 256 ?
a3ccf3ba-2649-4d40-8992-41d38dcab493 us-east-1e
UN 10.98.xxx.31 43.71 GB 256 ?
290a48fc-9efe-4e3d-9237-c363894c7b13 us-east-1d
UN 10.99.xxx.18739.76 GB 256 ?
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b us-east-1d
{noformat}
was (Author: jeffl):
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
4. nodetool status show that the nodes are having over 40GB data/load:
{noformat}
-- Address Load Tokens OwnsHost ID
Rack
UN 10.98.xxx.5343.62 GB 256 ?
1ab1fec7-383b-4801-b899-059ea0c18def us-east-1d
UN 10.182.xxx.24 46.53 GB 256 ?
07a9849b-73f2-4166-a8e9-bedd40a1f30e us-east-1e
UN 10.32.xxx.222 62.36 GB 256 ?
fb8c38fc-0eb1-41c2-96c3-fc597aeede10 us-east-1d
UN 10.66.xxx.48 45.03 GB 256 ?
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2 us-east-1c
UN 10.232.xxx.154 50.57 GB 256 ?
a3ccf3ba-2649-4d40-8992-41d38dcab493 us-east-1e
UN 10.98.xxx.31 43.71 GB 256 ?
290a48fc-9efe-4e3d-9237-c363894c7b13 us-east-1d
UN 10.99.79.18739.76 GB 256 ?
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b us-east-1d
{noformat}
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxre
[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309943#comment-14309943
]
Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:26 PM:
-
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
4. nodetool status show that the nodes are having over 40GB data/load:
{noformat}
-- Address Load Tokens OwnsHost ID
Rack
UN 10.98.xxx.5343.62 GB 256 ?
1ab1fec7-383b-4801-b899-059ea0c18def us-east-1d
UN 10.182.xxx.24 46.53 GB 256 ?
07a9849b-73f2-4166-a8e9-bedd40a1f30e us-east-1e
UN 10.32.xxx.222 62.36 GB 256 ?
fb8c38fc-0eb1-41c2-96c3-fc597aeede10 us-east-1d
UN 10.66.xxx.48 45.03 GB 256 ?
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2 us-east-1c
UN 10.232.xxx.154 50.57 GB 256 ?
a3ccf3ba-2649-4d40-8992-41d38dcab493 us-east-1e
UN 10.98.xxx.31 43.71 GB 256 ?
290a48fc-9efe-4e3d-9237-c363894c7b13 us-east-1d
UN 10.99.79.18739.76 GB 256 ?
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b us-east-1d
{noformat}
was (Author: jeffl):
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
4. nodetool status show that the nodes are having 40GB data/load on average:
{noformat}
-- Address Load Tokens OwnsHost ID
Rack
UN 10.98.xxx.5343.62 GB 256 ?
1ab1fec7-383b-4801-b899-059ea0c18def us-east-1d
UN 10.182.xxx.24 46.53 GB 256 ?
07a9849b-73f2-4166-a8e9-bedd40a1f30e us-east-1e
UN 10.32.xxx.222 62.36 GB 256 ?
fb8c38fc-0eb1-41c2-96c3-fc597aeede10 us-east-1d
UN 10.66.xxx.48 45.03 GB 256 ?
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2 us-east-1c
UN 10.232.xxx.154 50.57 GB 256 ?
a3ccf3ba-2649-4d40-8992-41d38dcab493 us-east-1e
UN 10.98.xxx.31 43.71 GB 256 ?
290a48fc-9efe-4e3d-9237-c363894c7b13 us-east-1d
UN 10.99.79.18739.76 GB 256 ?
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b us-east-1d
{noformat}
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.
[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309943#comment-14309943
]
Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:26 PM:
-
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
4. nodetool status show that the nodes are having 40GB data/load on average:
{noformat}
-- Address Load Tokens OwnsHost ID
Rack
UN 10.98.xxx.5343.62 GB 256 ?
1ab1fec7-383b-4801-b899-059ea0c18def us-east-1d
UN 10.182.xxx.24 46.53 GB 256 ?
07a9849b-73f2-4166-a8e9-bedd40a1f30e us-east-1e
UN 10.32.xxx.222 62.36 GB 256 ?
fb8c38fc-0eb1-41c2-96c3-fc597aeede10 us-east-1d
UN 10.66.xxx.48 45.03 GB 256 ?
45820ccb-ce05-4a1b-9fd5-23d1c1512dd2 us-east-1c
UN 10.232.xxx.154 50.57 GB 256 ?
a3ccf3ba-2649-4d40-8992-41d38dcab493 us-east-1e
UN 10.98.xxx.31 43.71 GB 256 ?
290a48fc-9efe-4e3d-9237-c363894c7b13 us-east-1d
UN 10.99.79.18739.76 GB 256 ?
8d9ca84a-4abe-40b4-b7a6-5424f84cab8b us-east-1d
{noformat}
was (Author: jeffl):
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor
[jira] [Comment Edited] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309943#comment-14309943
]
Jeff Liu edited comment on CASSANDRA-8723 at 2/6/15 9:24 PM:
-
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage with 1.7 reserved for jvm heap:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
was (Author: jeffl):
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib
[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309943#comment-14309943
]
Jeff Liu commented on CASSANDRA-8723:
-
hi Alan,
thanks for your relay and the answer is yes, we have been able to reproduce
it several times. The first cassandra version we observed this issue was 2.1.0,
and reproduced in 2.1.1 and 2.1.3.
1). java version 1.7.0_55
2). ubuntu 3.2.0-70-virtual . AWS c1.xlarge
3) process resource usage:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
21279 cassandr 20 0 21.1g 3.7g 23m S 149 55.1 12001:58 java
{noformat}
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassand
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309563#comment-14309563
]
Jeff Liu commented on CASSANDRA-8696:
-
[~yukim], [~benedict], any additional thoughts on this issue?
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
> at
> java.u
[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309559#comment-14309559
]
Jeff Liu commented on CASSANDRA-8723:
-
[~benedict] and the team, have you seen any others reporting this issue? This
is driving us nuts here validating cassandra 2.1.3. Without the memory
stability, we cannot even use this version for dev environment.
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(
[jira] [Commented] (CASSANDRA-8711) cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307704#comment-14307704
]
Jeff Liu commented on CASSANDRA-8711:
-
Here is some additional information:
When I don't have a cqlshrc file:
{noformat}
/home/jliu# cqlsh --ssl
Validation is enabled; SSL transport factory requires a valid certfile to be
specified. Please provide path to the certfile in [ssl] section as 'certfile'
option in /root/.cassandra/cqlshrc (or use [certfiles] section) or set
SSL_CERTFILE environment variable.
{noformat}
when I have a cqlshrc file(contents as pasted in description)
{noformat}
/home/jliu# cqlsh --ssl
Password:
Connection error: ('Unable to connect to any servers', {'localhost': error(8,
'ENOEXEC')})
{noformat}
> cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
>
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem (localhost_user1.pem) key have been
> verified to be working fine for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8711) cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307700#comment-14307700
]
Jeff Liu commented on CASSANDRA-8711:
-
Which port should be used?
> cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
>
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem (localhost_user1.pem) key have been
> verified to be working fine for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8711) cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307696#comment-14307696
]
Jeff Liu commented on CASSANDRA-8711:
-
hi [~mishail].
Yes. I did use the --ssl flag. Actually after I configure ~/.cassandra/cqlshrc
file and specify ssl configuration, cqlsh will pick the ssl configuration and
try to connect with ssl even without --ssl flag.
However, either one worked for me in terms of connecting to cassandra.
> cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
>
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem (localhost_user1.pem) key have been
> verified to be working fine for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8711) cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8711:
Summary: cassandra 2.1.2 cqlsh not able to connect when ssl client
encryption enabled (was: cassandra 2.1.2 ssl client encryption not working )
> cassandra 2.1.2 cqlsh not able to connect when ssl client encryption enabled
>
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem (localhost_user1.pem) key have been
> verified to be working fine for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8711) cassandra 2.1.2 ssl client encryption not working
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14305716#comment-14305716
]
Jeff Liu commented on CASSANDRA-8711:
-
After spending a little more time, I wrote some quick ruby code to test client
encryption/authentication and managed to get it working!
I think my problem was really the cqlsh not able to connect to cassandra nodes
with client encryption or authentication or both enabled.
here is my cqlshrc file contents:
{noformat}
[authentication]
username = cassandra
password = cassandra
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/private1.pem
validate = false ## Optional, true by default
{noformat}
the private1.pem is the private key that I was able to connect from client
(ruby script) with.
> cassandra 2.1.2 ssl client encryption not working
> --
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem (localhost_user1.pem) key have been
> verified to be working fine for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8723:
Attachment: cassandra.yaml
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
> Attachments: cassandra.yaml
>
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303751#comment-14303751
]
Jeff Liu commented on CASSANDRA-8723:
-
cassandra.yaml
{noformat}
# Cassandra storage config YAML
# NOTE:
# See http://wiki.apache.org/cassandra/StorageConfiguration for
# full explanations of configuration directives
# /NOTE
# The name of the cluster. This is mainly used to prevent machines in
# one logical cluster from joining another.
cluster_name: ch_cassandra
# This defines the number of tokens randomly assigned to this node on the ring
# The more tokens, relative to other nodes, the larger the proportion of data
# that this node will store. You probably want all nodes to have the same number
# of tokens assuming they have equal hardware capability.
#
# If you leave this unspecified, Cassandra will use the default of 1 token for
legacy compatibility,
# and will use the initial_token as described below.
#
# Specifying initial_token will override this setting.
#
# If you already have a cluster with 1 token per node, and wish to migrate to
# multiple tokens per node, see http://wiki.apache.org/cassandra/Operations
num_tokens: 256
# initial_token allows you to specify tokens manually. While you can use # it
with
# vnodes (num_tokens > 1, above) -- in which case you should provide a
# comma-separated list -- it's primarily used when adding nodes # to legacy
clusters
# that do not have vnodes enabled.
# initial_token:
initial_token:
# See http://wiki.apache.org/cassandra/HintedHandoff
hinted_handoff_enabled: true
# this defines the maximum amount of time a dead host will have hints
# generated. After it has been dead this long, new hints for it will not be
# created until it has been seen alive and gone down again.
max_hint_window_in_ms: 1080 # 3 hours
# Maximum throttle in KBs per second, per delivery thread. This will be
# reduced proportionally to the number of nodes in the cluster. (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously.)
hinted_handoff_throttle_in_kb: 1024
# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 2
# The following setting populates the page cache on memtable flush and
compaction
# WARNING: Enable this setting only when the whole node's data fits in memory.
# Defaults to: false
# populate_io_cache_on_flush: false
# Authentication backend, implementing IAuthenticator; used to identify users
# Out of the box, Cassandra provides
org.apache.cassandra.auth.{AllowAllAuthenticator,
# PasswordAuthenticator}.
#
# - AllowAllAuthenticator performs no checks - set it to disable authentication.
# - PasswordAuthenticator relies on username/password pairs to authenticate
# users. It keeps usernames and hashed passwords in system_auth.credentials
table.
# Please increase system_auth keyspace replication factor if you use this
authenticator.
authenticator: AllowAllAuthenticator
# Authorization backend, implementing IAuthorizer; used to limit access/provide
permissions
# Out of the box, Cassandra provides
org.apache.cassandra.auth.{AllowAllAuthorizer,
# CassandraAuthorizer}.
#
# - AllowAllAuthorizer allows any action to any user - set it to disable
authorization.
# - CassandraAuthorizer stores permissions in system_auth.permissions table.
Please
# increase system_auth keyspace replication factor if you use this authorizer.
authorizer: AllowAllAuthorizer
# Validity period for permissions cache (fetching permissions can be an
# expensive operation depending on the authorizer, CassandraAuthorizer is
# one example). Defaults to 2000, set to 0 to disable.
# Will be disabled automatically for AllowAllAuthorizer.
permissions_validity_in_ms: 2000
# The partitioner is responsible for distributing rows (by key) across
# nodes in the cluster. Any IPartitioner may be used, including your
# own as long as it is on the classpath. Out of the box, Cassandra
# provides org.apache.cassandra.dht.{Murmur3Partitioner, RandomPartitioner
# ByteOrderedPartitioner, OrderPreservingPartitioner (deprecated)}.
#
# - RandomPartitioner distributes rows across the cluster evenly by md5.
# This is the default prior to 1.2 and is retained for compatibility.
# - Murmur3Partitioner is similar to RandomPartioner but uses Murmur3_128
# Hash Function instead of md5. When in doubt, this is the best option.
# - ByteOrderedPartitioner orders rows lexically by key bytes. BOP allows
# scanning rows in key order, but the ordering can generate hot spots
# for sequential insertion workloads.
# - OrderPreservingPartitioner is an obsolete form of BOP, that stores
[jira] [Updated] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until process is killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8723:
Summary: Cassandra 2.1.2 Memory issue - java process memory usage
continuously increases until process is killed by OOM killer (was: Cassandra
2.1.2 Memory issue - java process memory usage continuously increases until
killed by OOM killer)
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until process is killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increase until killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8723:
Summary: Cassandra 2.1.2 Memory issue - java process memory usage
continuously increase until killed by OOM killer (was: Cassandra 2.1.2 Memory
issue - java process memory continuously increase until killed by OOM killer)
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increase until killed by OOM killer
> -
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory usage continuously increases until killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8723:
Summary: Cassandra 2.1.2 Memory issue - java process memory usage
continuously increases until killed by OOM killer (was: Cassandra 2.1.2 Memory
issue - java process memory usage continuously increase until killed by OOM
killer)
> Cassandra 2.1.2 Memory issue - java process memory usage continuously
> increases until killed by OOM killer
> --
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - java process memory continuously increase until killed by OOM killer
[
https://issues.apache.org/jira/browse/CASSANDRA-8723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8723:
Summary: Cassandra 2.1.2 Memory issue - java process memory continuously
increase until killed by OOM killer (was: Cassandra 2.1.2 Memory issue -
Continuously process memory increase until killed by OOM killer)
> Cassandra 2.1.2 Memory issue - java process memory continuously increase
> until killed by OOM killer
> ---
>
> Key: CASSANDRA-8723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> Issue:
> We have an on-going issue with cassandra nodes running with continuously
> increasing memory until killed by OOM.
> {noformat}
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
> process 13919 (java) score 911 or sacrifice child
> Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
> (java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
> {noformat}
> System Profile:
> cassandra version 2.1.2
> system: aws c1.xlarge instance with 8 cores, 7.1G memory.
> cassandra jvm:
> -Xms1792M -Xmx1792M -Xmn400M -Xss256k
> {noformat}
> java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
> -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
> -Dlogback.configurationFile=logback.xml
> -Dcassandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
> -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
> /etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
> -XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
> -XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
> org.apache.cassandra.service.CassandraDaemon
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8723) Cassandra 2.1.2 Memory issue - Continuously process memory increase until killed by OOM killer
Jeff Liu created CASSANDRA-8723:
---
Summary: Cassandra 2.1.2 Memory issue - Continuously process
memory increase until killed by OOM killer
Key: CASSANDRA-8723
URL: https://issues.apache.org/jira/browse/CASSANDRA-8723
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
Issue:
We have an on-going issue with cassandra nodes running with continuously
increasing memory until killed by OOM.
{noformat}
Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783481] Out of memory: Kill
process 13919 (java) score 911 or sacrifice child
Jan 29 10:15:41 cass-chisel19 kernel: [24533109.783557] Killed process 13919
(java) total-vm:18366340kB, anon-rss:6461472kB, file-rss:6684kB
{noformat}
System Profile:
cassandra version 2.1.2
system: aws c1.xlarge instance with 8 cores, 7.1G memory.
cassandra jvm:
-Xms1792M -Xmx1792M -Xmn400M -Xss256k
{noformat}
java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M
-Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+UseCondCardMark
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1421511249.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
-Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra
-Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid
-cp
/etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
-XX:HeapDumpPath=/var/lib/cassandra/java_1421511248.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1421511248.log
org.apache.cassandra.service.CassandraDaemon
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301596#comment-14301596
]
Jeff Liu commented on CASSANDRA-8689:
-
Surprised to know that there are so much information behind this blur java
assertion error. Thank you Benedict for the info. What is the impact of this
issue without fix? are in some kind of compacting danger ? I guess I'm really
asking for the criticality of this issue. Thanks.
> Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
> ---
>
> Key: CASSANDRA-8689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
> Fix For: 2.1.3
>
>
> After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
> following assertion error.
> {noformat}
> ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451
> CassandraDaemon.java:153 - Exception in thread
> Thread[IndexSummaryManager:1,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_45]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> {noformat}
> cassandra service is still running despite the issue. Node has total 8G
> memory with 2G allocated to heap. We are basically running read queries to
> retrieve data out of cassandra.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8711) cassandra 2.1.2 ssl client encryption not working
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8711:
Description:
I have been trying to setup client encryption on a three nodes 2.1.2 version
cassandra cluster and keep getting the following error:
{noformat}
Connection error: ('Unable to connect to any servers', {'localhost':
ConnectionShutdown('Connection is already closed',)})
{noformat}
I tried with both cqlsh and datatax python cassandra-driver and no luck to
login.
I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
{noformat}
[authentication]
username =
password =
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/localhost_user1.pem
validate = false ## Optional, true by default
{noformat}
my cassandra.yaml configuration related to client_encryptions:
{noformat}
client_encryption_options:
enabled: True
keystore: /etc/cassandra/conf/.keystore
keystore_password: cassnest
{noformat}
the keystore, truststore, cert/pem (localhost_user1.pem) key have been verified
to be working fine for datastax enterprise version.
was:
I have been trying to setup client encryption on a three nodes 2.1.2 version
cassandra cluster and keep getting the following error:
{noformat}
Connection error: ('Unable to connect to any servers', {'localhost':
ConnectionShutdown('Connection is already closed',)})
{noformat}
I tried with both cqlsh and datatax python cassandra-driver and no luck to
login.
I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
{noformat}
[authentication]
username =
password =
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/localhost_user1.pem
validate = false ## Optional, true by default
{noformat}
my cassandra.yaml configuration related to client_encryptions:
{noformat}
client_encryption_options:
enabled: True
keystore: /etc/cassandra/conf/.keystore
keystore_password: cassnest
{noformat}
the keystore, truststore, cert/pem key have been verified to be working fine
for datastax enterprise version.
> cassandra 2.1.2 ssl client encryption not working
> --
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem (localhost_user1.pem) key have been
> verified to be working fine for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8711) cassandra 2.1.2 ssl client encryption not working
Jeff Liu created CASSANDRA-8711:
---
Summary: cassandra 2.1.2 ssl client encryption not working
Key: CASSANDRA-8711
URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
I have been trying to setup client encryption on a three nodes 2.1.2 version
cassandra cluster and keep getting the following error:
{noformat}
Connection error: ('Unable to connect to any servers', {'localhost':
ConnectionShutdown('Connection is already closed',)})
{noformat}
I tried with both cqlsh and datatax python cassandra-driver and no luck to
login.
I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
{/noformat}
[authentication]
username =
password =
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/localhost_user1.pem
validate = false ## Optional, true by default
{/noformat}
my cassandra.yaml configuration related to client_encryptions:
{noformat}
client_encryption_options:
enabled: True
keystore: /etc/cassandra/conf/.keystore
keystore_password: cassnest
{noformat}
the keystore, truststore, cert/pem key have been verified to be working fine
for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8711) cassandra 2.1.2 ssl client encryption not working
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8711:
Description:
I have been trying to setup client encryption on a three nodes 2.1.2 version
cassandra cluster and keep getting the following error:
{noformat}
Connection error: ('Unable to connect to any servers', {'localhost':
ConnectionShutdown('Connection is already closed',)})
{noformat}
I tried with both cqlsh and datatax python cassandra-driver and no luck to
login.
I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
{noformat}
[authentication]
username =
password =
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/localhost_user1.pem
validate = false ## Optional, true by default
{noformat}
my cassandra.yaml configuration related to client_encryptions:
{noformat}
client_encryption_options:
enabled: True
keystore: /etc/cassandra/conf/.keystore
keystore_password: cassnest
{noformat}
the keystore, truststore, cert/pem key have been verified to be working fine
for datastax enterprise version.
was:
I have been trying to setup client encryption on a three nodes 2.1.2 version
cassandra cluster and keep getting the following error:
{noformat}
Connection error: ('Unable to connect to any servers', {'localhost':
ConnectionShutdown('Connection is already closed',)})
{noformat}
I tried with both cqlsh and datatax python cassandra-driver and no luck to
login.
I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
{noformat}
[authentication]
username =
password =
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/localhost_user1.pem
validate = false ## Optional, true by default
{/noformat}
my cassandra.yaml configuration related to client_encryptions:
{noformat}
client_encryption_options:
enabled: True
keystore: /etc/cassandra/conf/.keystore
keystore_password: cassnest
{noformat}
the keystore, truststore, cert/pem key have been verified to be working fine
for datastax enterprise version.
> cassandra 2.1.2 ssl client encryption not working
> --
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem key have been verified to be working fine
> for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8711) cassandra 2.1.2 ssl client encryption not working
[
https://issues.apache.org/jira/browse/CASSANDRA-8711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8711:
Description:
I have been trying to setup client encryption on a three nodes 2.1.2 version
cassandra cluster and keep getting the following error:
{noformat}
Connection error: ('Unable to connect to any servers', {'localhost':
ConnectionShutdown('Connection is already closed',)})
{noformat}
I tried with both cqlsh and datatax python cassandra-driver and no luck to
login.
I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
{noformat}
[authentication]
username =
password =
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/localhost_user1.pem
validate = false ## Optional, true by default
{/noformat}
my cassandra.yaml configuration related to client_encryptions:
{noformat}
client_encryption_options:
enabled: True
keystore: /etc/cassandra/conf/.keystore
keystore_password: cassnest
{noformat}
the keystore, truststore, cert/pem key have been verified to be working fine
for datastax enterprise version.
was:
I have been trying to setup client encryption on a three nodes 2.1.2 version
cassandra cluster and keep getting the following error:
{noformat}
Connection error: ('Unable to connect to any servers', {'localhost':
ConnectionShutdown('Connection is already closed',)})
{noformat}
I tried with both cqlsh and datatax python cassandra-driver and no luck to
login.
I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
{/noformat}
[authentication]
username =
password =
[connection]
hostname = localhost
port = 9160
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = /root/.cassandra/localhost_user1.pem
validate = false ## Optional, true by default
{/noformat}
my cassandra.yaml configuration related to client_encryptions:
{noformat}
client_encryption_options:
enabled: True
keystore: /etc/cassandra/conf/.keystore
keystore_password: cassnest
{noformat}
the keystore, truststore, cert/pem key have been verified to be working fine
for datastax enterprise version.
> cassandra 2.1.2 ssl client encryption not working
> --
>
> Key: CASSANDRA-8711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8711
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> I have been trying to setup client encryption on a three nodes 2.1.2 version
> cassandra cluster and keep getting the following error:
> {noformat}
> Connection error: ('Unable to connect to any servers', {'localhost':
> ConnectionShutdown('Connection (closed)> is already closed',)})
> {noformat}
> I tried with both cqlsh and datatax python cassandra-driver and no luck to
> login.
> I created /rooot/.cassandra/cqlshrc file for cqlsh settings, the content is:
> {noformat}
> [authentication]
> username =
> password =
> [connection]
> hostname = localhost
> port = 9160
> factory = cqlshlib.ssl.ssl_transport_factory
> [ssl]
> certfile = /root/.cassandra/localhost_user1.pem
> validate = false ## Optional, true by default
> {/noformat}
> my cassandra.yaml configuration related to client_encryptions:
> {noformat}
> client_encryption_options:
> enabled: True
> keystore: /etc/cassandra/conf/.keystore
> keystore_password: cassnest
> {noformat}
> the keystore, truststore, cert/pem key have been verified to be working fine
> for datastax enterprise version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297272#comment-14297272
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/29/15 6:21 PM:
--
It's getting worse today after having the "failed to create snapshot" error. 4
nodes of 6 are showing "UN" in nodetool status. I checked the system.log and
syslog and couldn't find anything that tells why the node was down. The java
cassandra process is also gone.
I tried to restart cassandra service on one node and watch system.log. Shortly
after the service start running, cassandra starts to throw "failed to create
snapshot" error for a while before it hung with no log output, nodetool status
show error:"error: No nodes present in the cluster. Has this node finished
starting up?". However the cassandra java process is still running in the
system.
{noformat}
ps -ef | grep cass
110 18634 1 19 18:01 ?00:01:39 java -ea
-javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M -Xmn400M
-XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
-XX:+UseParNewGC -XX
:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX
:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1422554504.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=719
9 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
-Dlogback.configurationFile=logback.xml -Dca
ssandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
-Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
/etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cas
sandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.ja
r:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar
:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassan
dra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassan
dra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cass
andra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/u
sr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
-XX:HeapDumpPath=/var/lib/cassandra/java_1422554504.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1422554504.log
org.apache.cassandra.service.CassandraDaemon
root 19317 19074 0 18:10 pts/100:00:00 grep --color=auto cass
#:/home/jliu# less /var/log/cassandra/system.log
{noformat}
{noformat}
#:/home/jliu# nodetool status
error: No nodes present in the cluster. Has this node finished starting up?
-- StackTrace --
java.lang.RuntimeException: No nodes present in the cluster. Has this node
finished starting up?
at
org.apache.cassandra.dht.RandomPartitioner.describeOwnership(RandomPartitioner.java:143)
at
org.apache.cassandra.service.StorageService.getOwnership(StorageService.java:3702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethod
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295881#comment-14295881
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/29/15 6:20 PM:
--
The replica nodes are not busy at all. Actually I have stopped all client
connection while trying to run a nodetool repair.
nodetool tpstats also shows that there isn't much activities going on.
{noformat}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 0 35441 0
0
RequestResponseStage 0 0 26940702 0
0
MutationStage 1 0 42298652 0
0
ReadRepairStage 0 0101 0
0
GossipStage 0 0 77379 0
0
CacheCleanupExecutor 0 0 0 0
0
AntiEntropyStage 0 0333 0
0
MigrationStage0 0 1664 0
0
ValidationExecutor0 0 51 0
0
CommitLogArchiver 0 0 0 0
0
MiscStage 0 0 0 0
0
MemtableFlushWriter 1 1853 0
0
MemtableReclaimMemory 0 0853 0
0
PendingRangeCalculator0 0 12 0
0
MemtablePostFlush 0 0 1365 0
0
CompactionExecutor2 9244 0
0
InternalResponseStage 0 0 1679 0
0
HintedHandoff 0 0 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 6067
MUTATION 8
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{noformat}
was (Author: jeffl):
The replica nodes are not busy at all. Actually I have stopped all client
connection while trying to run a nodetool repair.
nodetool tpstats also shows that there isn't much activities going on.
{noformat}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 0 35441 0
0
RequestResponseStage 0 0 26940702 0
0
MutationStage 1 0 42298652 0
0
ReadRepairStage 0 0101 0
0
GossipStage 0 0 77379 0
0
CacheCleanupExecutor 0 0 0 0
0
AntiEntropyStage 0 0333 0
0
MigrationStage0 0 1664 0
0
ValidationExecutor0 0 51 0
0
CommitLogArchiver 0 0 0 0
0
MiscStage 0 0 0 0
0
MemtableFlushWriter 1 1853 0
0
MemtableReclaimMemory 0 0853 0
0
PendingRangeCalculator0 0 12 0
0
MemtablePostFlush 0 0 1365 0
0
CompactionExecutor2 9244 0
0
InternalResponseStage 0 0 1679 0
0
HintedHandoff 0 0 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 6067
MUTATION 8
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{noformat}
ta
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295881#comment-14295881
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/29/15 6:19 PM:
--
The replica nodes are not busy at all. Actually I have stopped all client
connection while trying to run a nodetool repair.
nodetool tpstats also shows that there isn't much activities going on.
{noformat}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 0 35441 0
0
RequestResponseStage 0 0 26940702 0
0
MutationStage 1 0 42298652 0
0
ReadRepairStage 0 0101 0
0
GossipStage 0 0 77379 0
0
CacheCleanupExecutor 0 0 0 0
0
AntiEntropyStage 0 0333 0
0
MigrationStage0 0 1664 0
0
ValidationExecutor0 0 51 0
0
CommitLogArchiver 0 0 0 0
0
MiscStage 0 0 0 0
0
MemtableFlushWriter 1 1853 0
0
MemtableReclaimMemory 0 0853 0
0
PendingRangeCalculator0 0 12 0
0
MemtablePostFlush 0 0 1365 0
0
CompactionExecutor2 9244 0
0
InternalResponseStage 0 0 1679 0
0
HintedHandoff 0 0 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 6067
MUTATION 8
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{noformat}
tail -30 /var/log/cassandra/system.log where you can see the log suddenly
stopped.
{noformat}
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [AntiEntropySessions:38] 2015-01-28 19:47:39,746 CassandraDaemon.java:153
- Exception in thread Thread[AntiEntropySessions:38,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot
creation.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
... 3 common frames omitted
WARN [HintedHandoff:2] 2015-01-28 19:47:39,749 SliceQueryFilter.java:236 -
Read 0 live and 4525 tombstoned cells in system.hints (see
tombstone_warn_threshold). 128 columns was requested, slices=[-],
delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647}
INFO
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297272#comment-14297272
]
Jeff Liu commented on CASSANDRA-8696:
-
It's getting worse today after having the "failed to create snapshot" error. 4
nodes of 6 are showing "UN" in nodetool status. I checked the system.log and
syslog and couldn't find anything that tells why the node was down. The java
cassandra process is also gone.
I tried to restart cassandra service on one node and watch system.log. Shortly
after the service start running, cassandra starts to throw "failed to create
snapshot" error for a while before it hung with no log output, nodetool status
show error:"error: No nodes present in the cluster. Has this node finished
starting up?". However the cassandra java process is still running in the
system.
{noformat}
ps -ef | grep cass
110 18634 1 19 18:01 ?00:01:39 java -ea
-javaagent:/usr/share/cassandra/lib/jamm-0.2.8.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms1792M -Xmx1792M -Xmn400M
-XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
-XX:+UseParNewGC -XX
:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB -XX:+CMSClassUnloadingEnabled
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX
:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure -Xloggc:/var/log/cassandra/gc-1422554504.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=48M
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=719
9 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-javaagent:/usr/share/java/graphite-reporter-agent-1.0-SNAPSHOT.jar=graphiteServer=metrics-a.hq.nest.com;graphitePort=2003;graphitePollInt=60
-Dlogback.configurationFile=logback.xml -Dca
ssandra.logdir=/var/log/cassandra -Dcassandra.storagedir=
-Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
/etc/cassandra:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cas
sandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.ja
r:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.8.jar:/usr/share/cassandra/lib/javax.inject.jar
:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassan
dra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/metrics-graphite-2.2.0.jar:/usr/share/cassandra/lib/mx4j-tools.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassan
dra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/stringtemplate-4.0.2.jar:/usr/share/cass
andra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.2.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.2.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.5.jar:/u
sr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar:
-XX:HeapDumpPath=/var/lib/cassandra/java_1422554504.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1422554504.log
org.apache.cassandra.service.CassandraDaemon
root 19317 19074 0 18:10 pts/100:00:00 grep --color=auto cass
#:/home/jliu# less /var/log/cassandra/system.log
{noformat}
{noformat}
#:/home/jliu# nodetool status
error: No nodes present in the cluster. Has this node finished starting up?
-- StackTrace --
java.lang.RuntimeException: No nodes present in the cluster. Has this node
finished starting up?
at
org.apache.cassandra.dht.RandomPartitioner.describeOwnership(RandomPartitioner.java:143)
at
org.apache.cassandra.service.StorageService.getOwnership(StorageService.java:3702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295881#comment-14295881
]
Jeff Liu commented on CASSANDRA-8696:
-
The replica nodes are not busy at all. Actually I have stopped all client
connection while trying to run a nodetool repair.
nodetool tpstats also shows that there isn't much activities going on.
{noformat}
Pool NameActive Pending Completed Blocked All
time blocked
CounterMutationStage 0 0 0 0
0
ReadStage 0 0 35441 0
0
RequestResponseStage 0 0 26940702 0
0
MutationStage 1 0 42298652 0
0
ReadRepairStage 0 0101 0
0
GossipStage 0 0 77379 0
0
CacheCleanupExecutor 0 0 0 0
0
AntiEntropyStage 0 0333 0
0
MigrationStage0 0 1664 0
0
ValidationExecutor0 0 51 0
0
CommitLogArchiver 0 0 0 0
0
MiscStage 0 0 0 0
0
MemtableFlushWriter 1 1853 0
0
MemtableReclaimMemory 0 0853 0
0
PendingRangeCalculator0 0 12 0
0
MemtablePostFlush 0 0 1365 0
0
CompactionExecutor2 9244 0
0
InternalResponseStage 0 0 1679 0
0
HintedHandoff 0 0 13 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 6067
MUTATION 8
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{noformat}
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.ja
[jira] [Updated] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8696:
Summary: nodetool repair on cassandra 2.1.2 keyspaces return
java.lang.RuntimeException: Could not create snapshot (was: nodetool repair on
cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create
snapshot)
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> a
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295581#comment-14295581
]
Jeff Liu commented on CASSANDRA-8696:
-
nodetool repair returns: "Lost notification.You should check server log for
repair status of keyspace events"
# nodetool repair -pr
[2015-01-28 18:23:29,890] Starting repair command #1, repairing 256 ranges for
keyspace events (seq=true, full=true)
[2015-01-28 18:26:29,913] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:29:29,918] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:30:29,920] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:32:29,923] Lost notification. You should check server log for
repair status of keyspace events
> nodetool repair on cassandra 2.1.2 keyspaces
> returnjava.lang.RuntimeException: Could not create snapshot
>
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, big
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295581#comment-14295581
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/28/15 6:37 PM:
--
nodetool repair returns: "Lost notification.You should check server log for
repair status of keyspace events"
{noformat}
# nodetool repair -pr
[2015-01-28 18:23:29,890] Starting repair command #1, repairing 256 ranges for
keyspace events (seq=true, full=true)
[2015-01-28 18:26:29,913] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:29:29,918] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:30:29,920] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:32:29,923] Lost notification. You should check server log for
repair status of keyspace events
{noformat}
was (Author: jeffl):
nodetool repair returns: "Lost notification.You should check server log for
repair status of keyspace events"
{format}
# nodetool repair -pr
[2015-01-28 18:23:29,890] Starting repair command #1, repairing 256 ranges for
keyspace events (seq=true, full=true)
[2015-01-28 18:26:29,913] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:29:29,918] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:30:29,920] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:32:29,923] Lost notification. You should check server log for
repair status of keyspace events
{format}
> nodetool repair on cassandra 2.1.2 keyspaces
> returnjava.lang.RuntimeException: Could not create snapshot
>
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295581#comment-14295581
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/28/15 6:37 PM:
--
nodetool repair returns: "Lost notification.You should check server log for
repair status of keyspace events"
{format}
# nodetool repair -pr
[2015-01-28 18:23:29,890] Starting repair command #1, repairing 256 ranges for
keyspace events (seq=true, full=true)
[2015-01-28 18:26:29,913] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:29:29,918] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:30:29,920] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:32:29,923] Lost notification. You should check server log for
repair status of keyspace events
{format}
was (Author: jeffl):
nodetool repair returns: "Lost notification.You should check server log for
repair status of keyspace events"
# nodetool repair -pr
[2015-01-28 18:23:29,890] Starting repair command #1, repairing 256 ranges for
keyspace events (seq=true, full=true)
[2015-01-28 18:26:29,913] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:29:29,918] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:30:29,920] Lost notification. You should check server log for
repair status of keyspace events
[2015-01-28 18:32:29,923] Lost notification. You should check server log for
repair status of keyspace events
> nodetool repair on cassandra 2.1.2 keyspaces
> returnjava.lang.RuntimeException: Could not create snapshot
>
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
>
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295578#comment-14295578
]
Jeff Liu commented on CASSANDRA-8696:
-
[~yukim]
I continued to hop on 10.66.187.201 and run nodetool repair -pr on this node
and observed the similar failed to create snapshot error.
{/noformat}
INFO [RepairJobTask:15] 2015-01-28 18:29:25,378 StreamResultFuture.java:86 -
[Stream #93af91f0-a71b-11e4-820f-edcf08b136c7] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:5] 2015-01-28 18:29:25,379
StreamSession.java:213 - [Stream #93af91f0-a71b-11e4-820f-edcf08b136c7]
Starting streaming to /10.98.194.68
INFO [StreamConnectionEstablisher:5] 2015-01-28 18:29:25,384
StreamCoordinator.java:209 - [Stream #93af91f0-a71b-11e4-820f-edcf08b136c7,
ID#0] Beginning stream session with /10.98.194.68
INFO [STREAM-IN-/10.98.194.68] 2015-01-28 18:29:25,590
StreamResultFuture.java:166 - [Stream #93af91f0-a71b-11e4-820f-edcf08b136c7
ID#0] Prepare completed. Receiving 47 files(40043719 bytes), sending 1100
files(151144862 bytes)
ERROR [RepairJobTask:15] 2015-01-28 18:29:31,654 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.66.187.201
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [AntiEntropySessions:1] 2015-01-28 18:29:31,656 RepairSession.java:303 -
[repair #c35edd80-a71a-11e4-820f-edcf08b136c7] session completed with the
following error
java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
INFO [AntiEntropySessions:5] 2015-01-28 18:29:31,656 RepairSession.java:260 -
[repair #99efe970-a71b-11e4-820f-edcf08b136c7] new session: will sync
/10.66.187.201, /10.66.121.76, /10.97.9.110 on range
(1578063643453800039554350043004094,15825842670805971884119537801885597891]
for events.[column_categories, bigint0double, bigint0text, bigint0bigint,
bigint0int, bigint0boolean, dataset_catalog]
ERROR [AntiEntropySessions:1] 2015-01-28 18:29:31,658 CassandraDaemon.java:153
- Exception in thread Thread[AntiEntropySessions:1,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot
creation.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
... 3 common frames omitted
ERROR [RepairJobTask:3] 2015-01-28 18:29:41,664 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.66.187.201
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295578#comment-14295578
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/28/15 6:35 PM:
--
[~yukim]
I continued to hop on 10.66.187.201 and run nodetool repair -pr on this node
and observed the similar failed to create snapshot error.
{noformat}
INFO [RepairJobTask:15] 2015-01-28 18:29:25,378 StreamResultFuture.java:86 -
[Stream #93af91f0-a71b-11e4-820f-edcf08b136c7] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:5] 2015-01-28 18:29:25,379
StreamSession.java:213 - [Stream #93af91f0-a71b-11e4-820f-edcf08b136c7]
Starting streaming to /10.98.194.68
INFO [StreamConnectionEstablisher:5] 2015-01-28 18:29:25,384
StreamCoordinator.java:209 - [Stream #93af91f0-a71b-11e4-820f-edcf08b136c7,
ID#0] Beginning stream session with /10.98.194.68
INFO [STREAM-IN-/10.98.194.68] 2015-01-28 18:29:25,590
StreamResultFuture.java:166 - [Stream #93af91f0-a71b-11e4-820f-edcf08b136c7
ID#0] Prepare completed. Receiving 47 files(40043719 bytes), sending 1100
files(151144862 bytes)
ERROR [RepairJobTask:15] 2015-01-28 18:29:31,654 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.66.187.201
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [AntiEntropySessions:1] 2015-01-28 18:29:31,656 RepairSession.java:303 -
[repair #c35edd80-a71a-11e4-820f-edcf08b136c7] session completed with the
following error
java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
INFO [AntiEntropySessions:5] 2015-01-28 18:29:31,656 RepairSession.java:260 -
[repair #99efe970-a71b-11e4-820f-edcf08b136c7] new session: will sync
/10.66.187.201, /10.66.121.76, /10.97.9.110 on range
(1578063643453800039554350043004094,15825842670805971884119537801885597891]
for events.[column_categories, bigint0double, bigint0text, bigint0bigint,
bigint0int, bigint0boolean, dataset_catalog]
ERROR [AntiEntropySessions:1] 2015-01-28 18:29:31,658 CassandraDaemon.java:153
- Exception in thread Thread[AntiEntropySessions:1,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot
creation.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
... 3 common frames omitted
ERROR [RepairJobTask:3] 2015-01-28 18:29:41,664 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.66.187.201
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailu
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295547#comment-14295547
]
Jeff Liu commented on CASSANDRA-8696:
-
Interesting thing is that I haven't seen any repair error in node 10.97.9.110
system.log. From another post on serverfault website, Some other people
suggested that additional logs will show if manually running nodetool snapshot
on the problematic node. but I was able to successfully run nodetool snapshot
on node 10.97.9.110 without seeing any error log related to snapshot creation.
> nodetool repair on cassandra 2.1.2 keyspaces
> returnjava.lang.RuntimeException: Could not create snapshot
>
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.fai
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295552#comment-14295552
]
Jeff Liu commented on CASSANDRA-8696:
-
I take my previous reply back.
I do see similar error log of failing to create snapshot now.
{noformat}
INFO [RepairJobTask:14] 2015-01-28 18:15:07,253 Differencer.java:67 - [repair
#c4fa7700-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and /10.97.9.110
are consistent for bigint0bigint
INFO [RepairJobTask:15] 2015-01-28 18:15:07,254 Differencer.java:74 - [repair
#c4fa7700-a718-11e4-a4fa-21b31e869e75] Endpoints /10.66.187.201 and
/10.97.9.110 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,254 Differencer.java:74 - [repair
#c4fa7700-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and
/10.66.187.201 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,254 StreamingRepairTask.java:81 -
[repair #c4fa7700-a718-11e4-a4fa-21b31e869e75] Forwarding streaming repair of 1
ranges to /10.115.61.85 (to be streamed with /10.66.187.201)
INFO [AntiEntropyStage:1] 2015-01-28 18:15:07,256 RepairSession.java:171 -
[repair #634aaed0-a718-11e4-a4fa-21b31e869e75] Received merkle tree for
bigint0bigint from /10.97.9.110
INFO [RepairJobTask:14] 2015-01-28 18:15:07,257 Differencer.java:67 - [repair
#634aaed0-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and /10.97.9.110
are consistent for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,257 Differencer.java:74 - [repair
#634aaed0-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and
/10.66.187.201 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:15] 2015-01-28 18:15:07,257 Differencer.java:74 - [repair
#634aaed0-a718-11e4-a4fa-21b31e869e75] Endpoints /10.66.187.201 and
/10.97.9.110 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,257 StreamingRepairTask.java:81 -
[repair #634aaed0-a718-11e4-a4fa-21b31e869e75] Forwarding streaming repair of 1
ranges to /10.115.61.85 (to be streamed with /10.66.187.201)
INFO [RepairJobTask:15] 2015-01-28 18:15:07,330 StreamingRepairTask.java:68 -
[streaming task #c4fa7700-a718-11e4-a4fa-21b31e869e75] Performing streaming
repair of 1 ranges with /10.66.187.201
INFO [RepairJobTask:15] 2015-01-28 18:15:07,330 StreamingRepairTask.java:68 -
[streaming task #634aaed0-a718-11e4-a4fa-21b31e869e75] Performing streaming
repair of 1 ranges with /10.66.187.201
INFO [RepairJobTask:15] 2015-01-28 18:15:09,823 StreamResultFuture.java:86 -
[Stream #96c28930-a719-11e4-a4fa-21b31e869e75] Executing streaming plan for
Repair
INFO [RepairJobTask:15] 2015-01-28 18:15:09,823 StreamResultFuture.java:86 -
[Stream #96c28931-a719-11e4-a4fa-21b31e869e75] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:2] 2015-01-28 18:15:09,823
StreamSession.java:213 - [Stream #96c28930-a719-11e4-a4fa-21b31e869e75]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:3] 2015-01-28 18:15:09,824
StreamSession.java:213 - [Stream #96c28931-a719-11e4-a4fa-21b31e869e75]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:3] 2015-01-28 18:15:09,826
StreamCoordinator.java:209 - [Stream #96c28931-a719-11e4-a4fa-21b31e869e75,
ID#0] Beginning stream session with /10.66.187.201
INFO [StreamConnectionEstablisher:2] 2015-01-28 18:15:09,826
StreamCoordinator.java:209 - [Stream #96c28930-a719-11e4-a4fa-21b31e869e75,
ID#0] Beginning stream session with /10.66.187.201
ERROR [RepairJobTask:15] 2015-01-28 18:15:17,905 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.66.187.201
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [RepairJobTask:15] 2015-01-28 18:15:17,905 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295552#comment-14295552
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/28/15 6:17 PM:
--
I take my words saying no error log in my previous reply back.
I do see similar error log of failing to create snapshot now.
{noformat}
INFO [RepairJobTask:14] 2015-01-28 18:15:07,253 Differencer.java:67 - [repair
#c4fa7700-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and /10.97.9.110
are consistent for bigint0bigint
INFO [RepairJobTask:15] 2015-01-28 18:15:07,254 Differencer.java:74 - [repair
#c4fa7700-a718-11e4-a4fa-21b31e869e75] Endpoints /10.66.187.201 and
/10.97.9.110 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,254 Differencer.java:74 - [repair
#c4fa7700-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and
/10.66.187.201 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,254 StreamingRepairTask.java:81 -
[repair #c4fa7700-a718-11e4-a4fa-21b31e869e75] Forwarding streaming repair of 1
ranges to /10.115.61.85 (to be streamed with /10.66.187.201)
INFO [AntiEntropyStage:1] 2015-01-28 18:15:07,256 RepairSession.java:171 -
[repair #634aaed0-a718-11e4-a4fa-21b31e869e75] Received merkle tree for
bigint0bigint from /10.97.9.110
INFO [RepairJobTask:14] 2015-01-28 18:15:07,257 Differencer.java:67 - [repair
#634aaed0-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and /10.97.9.110
are consistent for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,257 Differencer.java:74 - [repair
#634aaed0-a718-11e4-a4fa-21b31e869e75] Endpoints /10.115.61.85 and
/10.66.187.201 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:15] 2015-01-28 18:15:07,257 Differencer.java:74 - [repair
#634aaed0-a718-11e4-a4fa-21b31e869e75] Endpoints /10.66.187.201 and
/10.97.9.110 have 1 range(s) out of sync for bigint0bigint
INFO [RepairJobTask:13] 2015-01-28 18:15:07,257 StreamingRepairTask.java:81 -
[repair #634aaed0-a718-11e4-a4fa-21b31e869e75] Forwarding streaming repair of 1
ranges to /10.115.61.85 (to be streamed with /10.66.187.201)
INFO [RepairJobTask:15] 2015-01-28 18:15:07,330 StreamingRepairTask.java:68 -
[streaming task #c4fa7700-a718-11e4-a4fa-21b31e869e75] Performing streaming
repair of 1 ranges with /10.66.187.201
INFO [RepairJobTask:15] 2015-01-28 18:15:07,330 StreamingRepairTask.java:68 -
[streaming task #634aaed0-a718-11e4-a4fa-21b31e869e75] Performing streaming
repair of 1 ranges with /10.66.187.201
INFO [RepairJobTask:15] 2015-01-28 18:15:09,823 StreamResultFuture.java:86 -
[Stream #96c28930-a719-11e4-a4fa-21b31e869e75] Executing streaming plan for
Repair
INFO [RepairJobTask:15] 2015-01-28 18:15:09,823 StreamResultFuture.java:86 -
[Stream #96c28931-a719-11e4-a4fa-21b31e869e75] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:2] 2015-01-28 18:15:09,823
StreamSession.java:213 - [Stream #96c28930-a719-11e4-a4fa-21b31e869e75]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:3] 2015-01-28 18:15:09,824
StreamSession.java:213 - [Stream #96c28931-a719-11e4-a4fa-21b31e869e75]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:3] 2015-01-28 18:15:09,826
StreamCoordinator.java:209 - [Stream #96c28931-a719-11e4-a4fa-21b31e869e75,
ID#0] Beginning stream session with /10.66.187.201
INFO [StreamConnectionEstablisher:2] 2015-01-28 18:15:09,826
StreamCoordinator.java:209 - [Stream #96c28930-a719-11e4-a4fa-21b31e869e75,
ID#0] Beginning stream session with /10.66.187.201
ERROR [RepairJobTask:15] 2015-01-28 18:15:17,905 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.66.187.201
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [RepairJobTask:15] 2015-01-28 18:15:17,905 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.Mes
[jira] [Comment Edited] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295547#comment-14295547
]
Jeff Liu edited comment on CASSANDRA-8696 at 1/28/15 6:15 PM:
--
Interesting thing is that I haven't seen any repair error in node 10.97.9.110
system.log. From another post on serverfault website, another people suggested
that additional logs will show if manually running nodetool snapshot on the
problematic node. but I was able to successfully run nodetool snapshot on node
10.97.9.110 without seeing any error log related to snapshot creation.
was (Author: jeffl):
Interesting thing is that I haven't seen any repair error in node 10.97.9.110
system.log. From another post on serverfault website, Some other people
suggested that additional logs will show if manually running nodetool snapshot
on the problematic node. but I was able to successfully run nodetool snapshot
on node 10.97.9.110 without seeing any error log related to snapshot creation.
> nodetool repair on cassandra 2.1.2 keyspaces
> returnjava.lang.RuntimeException: Could not create snapshot
>
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,1
[jira] [Created] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces returnjava.lang.RuntimeException: Could not create snapshot
Jeff Liu created CASSANDRA-8696:
---
Summary: nodetool repair on cassandra 2.1.2 keyspaces
returnjava.lang.RuntimeException: Could not create snapshot
Key: CASSANDRA-8696
URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
Project: Cassandra
Issue Type: Bug
Reporter: Jeff Liu
When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
throw java exceptions: cannot create snapshot.
the error log from system.log:
{noformat}
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
files(632105 bytes)
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
Session with /10.97.9.110 is complete
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
All sessions completed
INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
[Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
Repair
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Starting streaming to /10.66.187.201
INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
ID#0] Beginning stream session with /10.66.187.201
INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
files(632105 bytes)
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
Session with /10.66.187.201 is complete
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
All sessions completed
INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
streaming task succeed, returning response to /10.98.194.68
ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
at
org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260 -
[repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
/10.98.194.68, /10.66.187.201, /10.226.218.135 on range
(12817179804668051873746972069086
2638799,12863540308359254031520865977436165] for events.[bigint0text,
bigint0boolean, bigint0int, dataset_catalog, column_categories, bigint0double,
bigint0bigint]
ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303 -
[repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
following error
java.io.IOException: Failed during snapshot creation.
at
org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,446 CassandraDaemon.java:153
- Exception in thread Thread[AntiEntropySessions:5,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot
creation.
at com.goog
[jira] [Updated] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8689:
Description:
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451 CassandraDaemon.java:153
- Exception in thread Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G memory
with 2G allocated to heap. We are basically running read queries to retrieve
data out of cassandra.
was:
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451 CassandraDaemon.java:153
- Exception in thread Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G memory
with 2G allocated to heap.
> Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
> -
[jira] [Updated] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
[
https://issues.apache.org/jira/browse/CASSANDRA-8689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Liu updated CASSANDRA-8689:
Description:
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
{noformat}
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451 CassandraDaemon.java:153
- Exception in thread Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
{noformat}
cassandra service is still running despite the issue. Node has total 8G memory
with 2G allocated to heap.
was:
After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the
following assertion error.
ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451 CassandraDaemon.java:153
- Exception in thread Thread[IndexSummaryManager:1,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_45]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
> ---
>
> Key: CASSANDRA-8689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8689
> Project: Cassandra
> Issue Type: Bug
>
[jira] [Created] (CASSANDRA-8689) Assertion error in 2.1.2: ERROR [IndexSummaryManager:1]
Jeff Liu created CASSANDRA-8689: --- Summary: Assertion error in 2.1.2: ERROR [IndexSummaryManager:1] Key: CASSANDRA-8689 URL: https://issues.apache.org/jira/browse/CASSANDRA-8689 Project: Cassandra Issue Type: Bug Reporter: Jeff Liu After upgrading a 6 nodes cassandra from 2.1.0 to 2.1.2, start getting the following assertion error. ERROR [IndexSummaryManager:1] 2015-01-26 20:55:40,451 CassandraDaemon.java:153 - Exception in thread Thread[IndexSummaryManager:1,1,main] java.lang.AssertionError: null at org.apache.cassandra.io.util.Memory.size(Memory.java:307) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.IndexSummary.getOffHeapSize(IndexSummary.java:192) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.SSTableReader.getIndexSummaryOffHeapSize(SSTableReader.java:1070) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:292) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77) ~[apache-cassandra-2.1.2.jar:2.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_45] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
