[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600657#comment-14600657 ] Study Hsueh commented on CASSANDRA-9607: My colleague said he performed the following queries before the whole cluster went down. 1. SELECT * FROM ginger.supply_ad_log - timeout 2. SELECT rmaxSpaceId FROM Supply_Ad_Log WHERE rmaxSpaceId = ? AND salesChannel = ? AND hourstamp = ? - query was hanged and cluster did not have responsed The client tool was gocql and he did not specify `pagesize` (default: 0). Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Tyler Hobbs Priority: Critical Fix For: 2.1.x, 2.2.x Attachments: GC_state.png, cassandra.yaml, client_blocked_thread.png, cpu_profile.png, dump.tdump, load.png, log.zip, schema.zip, vm_monitor.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14599684#comment-14599684 ] Robbie Strickland commented on CASSANDRA-9607: -- 2.1.7 with the patch works great. Thanks! Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Tyler Hobbs Priority: Critical Fix For: 2.1.x, 2.2.x Attachments: GC_state.png, cassandra.yaml, client_blocked_thread.png, cpu_profile.png, dump.tdump, load.png, log.zip, schema.zip, vm_monitor.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600666#comment-14600666 ] Study Hsueh commented on CASSANDRA-9607: My colleague have repeated the query in 2.1.3 again, and the cluster went down again. So the root cause should be the query. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Tyler Hobbs Priority: Critical Fix For: 2.1.x, 2.2.x Attachments: GC_state.png, cassandra.yaml, client_blocked_thread.png, cpu_profile.png, dump.tdump, load.png, log.zip, schema.zip, vm_monitor.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598087#comment-14598087 ] Benedict commented on CASSANDRA-9607: - [~thobbs]: if you look at the thread stack snapshot attached to the heap dump, it looks like it _is_ being performed with paging. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Tyler Hobbs Priority: Critical Fix For: 2.1.x, 2.2.x Attachments: GC_state.png, cassandra.yaml, client_blocked_thread.png, cpu_profile.png, dump.tdump, load.png, log.zip, schema.zip, vm_monitor.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[
https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598092#comment-14598092
]
Benedict commented on CASSANDRA-9607:
-
{noformat}
Object / Stack Frame
|Name| Shallow Heap | Retained Heap |Context Class
Loader|Is Daemon
---
at sun.misc.Unsafe.park(ZJ)V (Native Method)
|| | |
|
at java.util.concurrent.locks.LockSupport.parkNanos(J)V (LockSupport.java:349)
|| | |
|
at
org.apache.cassandra.utils.concurrent.WaitQueue$AbstractSignal.awaitUntil(J)Z
(WaitQueue.java:307)
|| | |
|
at
org.apache.cassandra.utils.concurrent.SimpleCondition.await(JLjava/util/concurrent/TimeUnit;)Z
(SimpleCondition.java:63)
|| | ||
at
org.apache.cassandra.service.ReadCallback.await(JLjava/util/concurrent/TimeUnit;)Z
(ReadCallback.java:90)
|| | ||
at
org.apache.cassandra.service.AbstractReadExecutor$SpeculatingReadExecutor.maybeTryAdditionalReplicas()V
(AbstractReadExecutor.java:271)
|
| | ||
at
org.apache.cassandra.service.StorageProxy.fetchRows(Ljava/util/List;Lorg/apache/cassandra/db/ConsistencyLevel;)Ljava/util/List;
(StorageProxy.java:1337)
|| |
||
at
org.apache.cassandra.service.StorageProxy.readRegular(Ljava/util/List;Lorg/apache/cassandra/db/ConsistencyLevel;)Ljava/util/List;
(StorageProxy.java:1273)
|| |
||
at
org.apache.cassandra.service.StorageProxy.read(Ljava/util/List;Lorg/apache/cassandra/db/ConsistencyLevel;Lorg/apache/cassandra/service/ClientState;)Ljava/util/List;
(StorageProxy.java:1196)
|| | ||
at
org.apache.cassandra.service.pager.SliceQueryPager.queryNextPage(ILorg/apache/cassandra/db/ConsistencyLevel;Z)Ljava/util/List;
(SliceQueryPager.java:90)
|| |
||
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(I)Ljava/util/List;
(AbstractQueryPager.java:85)
|| | ||
at
org.apache.cassandra.service.pager.SliceQueryPager.fetchPage(I)Ljava/util/List;
(SliceQueryPager.java:38)
|| | ||
at
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[
https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598116#comment-14598116
]
Tyler Hobbs commented on CASSANDRA-9607:
That might be a query on a different table, or maybe paging is being performed
with the max possible page size. The RangeSliceCommands on {{supply_ad_log}}
all have a {{maxResults}} of Integer.MAX_VALUE, and their predicates (slices)
have the same limit.
Get high load after upgrading from 2.1.3 to cassandra 2.1.6
---
Key: CASSANDRA-9607
URL: https://issues.apache.org/jira/browse/CASSANDRA-9607
Project: Cassandra
Issue Type: Bug
Environment: OS:
CentOS 6 * 4
Ubuntu 14.04 * 2
JDK: Oracle JDK 7, Oracle JDK 8
VM: Azure VM Standard A3 * 6
RAM: 7 GB
Cores: 4
Reporter: Study Hsueh
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.x, 2.2.x
Attachments: GC_state.png, cassandra.yaml, client_blocked_thread.png,
cpu_profile.png, dump.tdump, load.png, log.zip, schema.zip, vm_monitor.png
After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my
cassandra cluster grows from 0.x~1.x to 3.x~6.x.
What kind of additional information should I provide for this problem?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[
https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598061#comment-14598061
]
Tyler Hobbs commented on CASSANDRA-9607:
[~study] what sort of queries do you perform on the {{ginger.supply_ad_log}}
table? Just looking at the heap dump, it looks like you're doing something
like {{SELECT * FROM ginger.supply_ad_log}} without paging and with no limit on
the query. The results for just one small token range include 150,000+ cells.
Get high load after upgrading from 2.1.3 to cassandra 2.1.6
---
Key: CASSANDRA-9607
URL: https://issues.apache.org/jira/browse/CASSANDRA-9607
Project: Cassandra
Issue Type: Bug
Environment: OS:
CentOS 6 * 4
Ubuntu 14.04 * 2
JDK: Oracle JDK 7, Oracle JDK 8
VM: Azure VM Standard A3 * 6
RAM: 7 GB
Cores: 4
Reporter: Study Hsueh
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.x, 2.2.x
Attachments: GC_state.png, cassandra.yaml, client_blocked_thread.png,
cpu_profile.png, dump.tdump, load.png, log.zip, schema.zip, vm_monitor.png
After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my
cassandra cluster grows from 0.x~1.x to 3.x~6.x.
What kind of additional information should I provide for this problem?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598542#comment-14598542 ] T Jake Luciani commented on CASSANDRA-9607: --- This issue looks like CASSANDRA-9637, can you try that patch with your test [~rstrickland]? Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Tyler Hobbs Priority: Critical Fix For: 2.1.x, 2.2.x Attachments: GC_state.png, cassandra.yaml, client_blocked_thread.png, cpu_profile.png, dump.tdump, load.png, log.zip, schema.zip, vm_monitor.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596022#comment-14596022 ] Robbie Strickland commented on CASSANDRA-9607: -- [~tjake] I'll go wrangle someone from devops to get me set up to do that. BTW, we did just downgrade the production cluster from 2.1.5 to 2.1.4, and our Spark jobs all work now. Two weeks of downtime. Yay. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Tyler Hobbs Priority: Critical Fix For: 2.1.x, 2.2.0 rc2 Attachments: cassandra.yaml, load.png, log.zip, schema.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[
https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595975#comment-14595975
]
Robbie Strickland commented on CASSANDRA-9607:
--
I did some further testing on this, to at least isolate the affected versions
and nail down steps to reproduce. I started with a test cluster, 3 x
i2.4xlarge instances running 2.1.4 and Spark 1.2.2. I loaded a few sstables
(using sstableloader) from one of my problematic production tables, and each
test involved the following simple statement in the Spark shell:
{code}
sc.cassandraTable(prod_analytics_events,
profileevents).take(100).foreach(println)
{code}
I then upgraded to 2.1.6, loaded about 100G of data, and downgraded back to
2.1.4. The net result is that 2.1.4 works with both the small and larger data
sets, while 2.1.5 and 2.1.6 work only with only the smaller data sets. The
larger set caused the hang in both cases.
Get high load after upgrading from 2.1.3 to cassandra 2.1.6
---
Key: CASSANDRA-9607
URL: https://issues.apache.org/jira/browse/CASSANDRA-9607
Project: Cassandra
Issue Type: Bug
Environment: OS:
CentOS 6 * 4
Ubuntu 14.04 * 2
JDK: Oracle JDK 7, Oracle JDK 8
VM: Azure VM Standard A3 * 6
RAM: 7 GB
Cores: 4
Reporter: Study Hsueh
Assignee: Tyler Hobbs
Priority: Critical
Fix For: 2.1.x, 2.2.0 rc2
Attachments: cassandra.yaml, load.png, log.zip, schema.zip
After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my
cassandra cluster grows from 0.x~1.x to 3.x~6.x.
What kind of additional information should I provide for this problem?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595997#comment-14595997 ] T Jake Luciani commented on CASSANDRA-9607: --- [~rstrickland] can you profile a node (with VisualVM) in both cases so we can see what it's busy working on? Just attach screenshots of the sorted profile results Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: T Jake Luciani Priority: Critical Fix For: 2.1.x, 2.2.0 rc2 Attachments: cassandra.yaml, load.png, log.zip, schema.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593703#comment-14593703 ] Benedict commented on CASSANDRA-9607: - Ok, so on further investigation this is *not* caused by CASSANDRA-9549. It appears that the nodes are dying performing range slice queries, but it also *appears* that these queries are utilising paging. [~thobbs] [~blerer] [~iamaleksey], has anything changed around these subsystems since 2.1.3 that might cause them to misbehave? I would say we should delay 2.1.7 until we figure this out, but CASSANDRA-9549 is really pretty urgent. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Benedict Priority: Critical Attachments: cassandra.yaml, load.png, log.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593716#comment-14593716 ] Robbie Strickland commented on CASSANDRA-9607: -- This is interesting, because we are currently troubleshooting these exact symptoms on our analytics cluster when querying large tables using Spark. We had suspected sstable corruption, since some tables do work. But size appears to matter. Further, as I read your comment we had just finished loading one of the problematic tables into a test cluster running 2.1.4, and the same Spark job runs problem free. I am quite sure there's a correlation here. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Benedict Priority: Critical Attachments: cassandra.yaml, load.png, log.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593723#comment-14593723 ] Tyler Hobbs commented on CASSANDRA-9607: Sure, your schema might help us figure out what's going on. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Benedict Priority: Critical Attachments: cassandra.yaml, load.png, log.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593719#comment-14593719 ] Study Hsueh commented on CASSANDRA-9607: We do not use static columns. Do you need my schema for analysis? Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Benedict Priority: Critical Attachments: cassandra.yaml, load.png, log.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593711#comment-14593711 ] Tyler Hobbs commented on CASSANDRA-9607: Perhaps CASSANDRA-8502? [~study] do you have any tables with static columns? Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Benedict Priority: Critical Attachments: cassandra.yaml, load.png, log.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593753#comment-14593753 ] Study Hsueh commented on CASSANDRA-9607: Ok, I have uploaded my schema. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Assignee: Benedict Priority: Critical Attachments: cassandra.yaml, load.png, log.zip, schema.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[
https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591977#comment-14591977
]
Robbie Strickland commented on CASSANDRA-9607:
--
Yes it does. It seems from our observation to be related to 1) our heavy use
of DTCS combined with
[CASSANDRA-9549|https://issues.apache.org/jira/browse/CASSANDRA-9549] and 2)
severe GC pauses (such that it was GCing constantly).
We were able to make things stable by moving to G1 with the following
modifications:
{{code}}
#JVM_OPTS=$JVM_OPTS -Xmn${HEAP_NEWSIZE}
JVM_OPTS=$JVM_OPTS -XX:+UseG1GC
JVM_OPTS=$JVM_OPTS -XX:MaxGCPauseMillis=1000
JVM_OPTS=$JVM_OPTS -XX:+UseTLAB -XX:+ResizeTLAB
JVM_OPTS=$JVM_OPTS -XX:-UseBiasedLocking
{{code}}
This is still a work in progress, but it has allowed us to reach a stable state.
Get high load after upgrading from 2.1.3 to cassandra 2.1.6
---
Key: CASSANDRA-9607
URL: https://issues.apache.org/jira/browse/CASSANDRA-9607
Project: Cassandra
Issue Type: Bug
Environment: OS:
CentOS 6 * 4
Ubuntu 14.04 * 2
JDK: Oracle JDK 7
Reporter: Study Hsueh
Priority: Critical
Attachments: load.png
After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my
cassandra cluster grows from 0.x~1.x to 3.x~6.x.
What kind of additional information should I provide for this problem?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591843#comment-14591843 ] Jonathan Ellis commented on CASSANDRA-9607: --- Sound familiar [~rstrickland]? Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7 Reporter: Study Hsueh Priority: Critical Attachments: load.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592186#comment-14592186 ] Benedict commented on CASSANDRA-9607: - bq. I had uploaded heap dump when OOM occurred: Ah, you hadn't mentioned OOM. In that case it is _highly likely_ you are experiencing CASSANDRA-9549, which will be fixed in 2.1.7 released shortly, or you can run the patch version posted to that ticket as [~rstrickland] has. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Priority: Critical Attachments: cassandra.yaml, load.png, log.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14592306#comment-14592306 ] Study Hsueh commented on CASSANDRA-9607: Thanks, but I think we have no plan to try 2.1.6 and 2.1.7 in the near future. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7, Oracle JDK 8 VM: Azure VM Standard A3 * 6 RAM: 7 GB Cores: 4 Reporter: Study Hsueh Priority: Critical Attachments: cassandra.yaml, load.png, log.zip After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[
https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591503#comment-14591503
]
Benedict commented on CASSANDRA-9607:
-
I believe there were issues with compactions not being submitted in 2.1.3, so
the upgrade may have triggered your compaction workload to start. In this case,
turning down your concurrent_compactions settings in the yaml would resolve the
problem for you, and let it catch up with compactions.
If that isn't the problem, we would need, preferably, a CPU profile from an
attached debugger, or alternatively a sequence of {{jstack}} and {{top -H -b
-n1}} outputs to analyze. Along with your cassandra.yaml and log file.
Get high load after upgrading from 2.1.3 to cassandra 2.1.6
---
Key: CASSANDRA-9607
URL: https://issues.apache.org/jira/browse/CASSANDRA-9607
Project: Cassandra
Issue Type: Bug
Environment: OS:
CentOS 6 * 4
Ubuntu 14.04 * 2
JDK: Oracle JDK 7
Reporter: Study Hsueh
Priority: Critical
Attachments: load.png
After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my
cassandra cluster grows from 0.x~1.x to 3.x~6.x.
What kind of additional information should I provide for this problem?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589589#comment-14589589 ] Study Hsueh commented on CASSANDRA-9607: This issues cause all of nodes downs. Get high load after upgrading from 2.1.3 to cassandra 2.1.6 --- Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7 Reporter: Study Hsueh Priority: Critical Attachments: load.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
