[jira] [Updated] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-13019:
Fix Version/s: 4.0-beta
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x, 4.0-beta
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225253#comment-17225253
]
Marcus Eriksson commented on CASSANDRA-13019:
-
rebased on current trunk:
https://github.com/krummas/cassandra/commits/jeff/13019
cci run:
https://app.circleci.com/pipelines/github/krummas/cassandra?branch=jeff%2F13019
could someone sanity check before I commit?
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x, 4.0-beta
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225253#comment-17225253
]
Marcus Eriksson edited comment on CASSANDRA-13019 at 11/3/20, 9:12 AM:
---
rebased on current trunk:
https://github.com/krummas/cassandra/commits/jeff/13019
cci run:
https://app.circleci.com/pipelines/github/krummas/cassandra?branch=jeff%2F13019
Could someone sanity check before I commit? Only significant conflict was due
to CASSANDRA-15143 in {{FileUtils.java}}
was (Author: krummas):
rebased on current trunk:
https://github.com/krummas/cassandra/commits/jeff/13019
cci run:
https://app.circleci.com/pipelines/github/krummas/cassandra?branch=jeff%2F13019
could someone sanity check before I commit?
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x, 4.0-beta
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16114) Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
[
https://issues.apache.org/jira/browse/CASSANDRA-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225263#comment-17225263
]
Benjamin Lerer commented on CASSANDRA-16114:
The problem is affecting previous version too. Backporting the patch for those.
> Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
>
>
> Key: CASSANDRA-16114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16114
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Build:
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/494/workflows/b3765545-7b09-48dd-85ff-830c4f348329/jobs/2681
> {code}
> java.lang.ClassCastException:
> org.apache.cassandra.transport.messages.ResultMessage$Void cannot be cast to
> org.apache.cassandra.transport.messages.ResultMessage$SchemaChange
> at
> org.apache.cassandra.cql3.CQLTester.assertLastSchemaChange(CQLTester.java:916)
> at
> org.apache.cassandra.cql3.validation.entities.UFTest.testSchemaChange(UFTest.java:94)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16114) Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
[
https://issues.apache.org/jira/browse/CASSANDRA-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225263#comment-17225263
]
Benjamin Lerer edited comment on CASSANDRA-16114 at 11/3/20, 12:15 PM:
---
The problem is affecting previous versions too. Backporting the patch for those.
was (Author: blerer):
The problem is affecting previous version too. Backporting the patch for those.
> Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
>
>
> Key: CASSANDRA-16114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16114
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Build:
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/494/workflows/b3765545-7b09-48dd-85ff-830c4f348329/jobs/2681
> {code}
> java.lang.ClassCastException:
> org.apache.cassandra.transport.messages.ResultMessage$Void cannot be cast to
> org.apache.cassandra.transport.messages.ResultMessage$SchemaChange
> at
> org.apache.cassandra.cql3.CQLTester.assertLastSchemaChange(CQLTester.java:916)
> at
> org.apache.cassandra.cql3.validation.entities.UFTest.testSchemaChange(UFTest.java:94)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16114) Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
[
https://issues.apache.org/jira/browse/CASSANDRA-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225334#comment-17225334
]
Benjamin Lerer commented on CASSANDRA-16114:
||Branch PR|| CI ||
|[cassandra-2.2|https://github.com/apache/cassandra/pull/803]|[j8|https://app.circleci.com/pipelines/github/blerer/cassandra/53/workflows/61ff3832-f724-4f56-8115-6d78e3c6903c]|
|[cassandra-3.0|https://github.com/apache/cassandra/pull/804]|[j8|https://app.circleci.com/pipelines/github/blerer/cassandra/54/workflows/efc0b9c5-95eb-4e9b-9e62-0df834bcf0f9]|
|[cassandra-3.11|https://github.com/apache/cassandra/pull/805]|[j8|https://app.circleci.com/pipelines/github/blerer/cassandra/51/workflows/969b6c8f-a2c9-405e-a3b1-79ce19000698]|
|[trunk|https://github.com/apache/cassandra/pull/806]|[j8|https://app.circleci.com/pipelines/github/blerer/cassandra/56/workflows/fce458af-2cb1-476a-934b-e4276e0af16b],
[j11|https://app.circleci.com/pipelines/github/blerer/cassandra/56/workflows/9f6f8b58-60a0-4213-9c73-f87e4c809d37]|
> Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
>
>
> Key: CASSANDRA-16114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16114
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Build:
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/494/workflows/b3765545-7b09-48dd-85ff-830c4f348329/jobs/2681
> {code}
> java.lang.ClassCastException:
> org.apache.cassandra.transport.messages.ResultMessage$Void cannot be cast to
> org.apache.cassandra.transport.messages.ResultMessage$SchemaChange
> at
> org.apache.cassandra.cql3.CQLTester.assertLastSchemaChange(CQLTester.java:916)
> at
> org.apache.cassandra.cql3.validation.entities.UFTest.testSchemaChange(UFTest.java:94)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16114) Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
[
https://issues.apache.org/jira/browse/CASSANDRA-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225346#comment-17225346
]
Benjamin Lerer commented on CASSANDRA-16114:
CI looks good for the different branches.
> Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
>
>
> Key: CASSANDRA-16114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16114
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Build:
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/494/workflows/b3765545-7b09-48dd-85ff-830c4f348329/jobs/2681
> {code}
> java.lang.ClassCastException:
> org.apache.cassandra.transport.messages.ResultMessage$Void cannot be cast to
> org.apache.cassandra.transport.messages.ResultMessage$SchemaChange
> at
> org.apache.cassandra.cql3.CQLTester.assertLastSchemaChange(CQLTester.java:916)
> at
> org.apache.cassandra.cql3.validation.entities.UFTest.testSchemaChange(UFTest.java:94)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14477) The check of num_tokens against the length of inital_token in the yaml triggers unexpectedly
[ https://issues.apache.org/jira/browse/CASSANDRA-14477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225381#comment-17225381 ] Stefan Miklosovic commented on CASSANDRA-14477: --- [~mck] changed applied, the links with PRs are still valid, there is a link to each build there. > The check of num_tokens against the length of inital_token in the yaml > triggers unexpectedly > > > Key: CASSANDRA-14477 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14477 > Project: Cassandra > Issue Type: Bug > Components: Local/Config >Reporter: Vincent White >Assignee: Stefan Miklosovic >Priority: Low > Time Spent: 0.5h > Remaining Estimate: 0h > > In CASSANDRA-10120 we added a check that compares num_tokens against the > number of tokens supplied in the yaml via initial_token. From my reading of > CASSANDRA-10120 it was to prevent cassandra starting if the yaml contained > contradictory values for num_tokens and initial_tokens which should help > prevent misconfiguration via human error. The current behaviour appears to > differ slightly in that it performs this comparison regardless of whether > num_tokens is included in the yaml or not. Below are proposed patches to only > perform the check if both options are present in the yaml. > ||Branch|| > |[3.0.x|https://github.com/apache/cassandra/compare/cassandra-3.0...vincewhite:num_tokens_30]| > |[3.x|https://github.com/apache/cassandra/compare/cassandra-3.11...vincewhite:num_tokens_test_1_311]| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated (f106ef0 -> 5a87472)
This is an automated email from the ASF dual-hosted git repository. blerer pushed a change to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from f106ef0 Fix invalid cell value skipping when reading from disk add ec2f2e6 Avoid storing the last schema change in CQLTester add 5a87472 Merge branch cassandra-2.2 into cassandra-3.0 No new revisions were added by this update. Summary of changes: test/unit/org/apache/cassandra/cql3/CQLTester.java | 60 +++- .../cassandra/cql3/validation/entities/UFTest.java | 81 +++--- .../validation/operations/AggregationTest.java | 76 ++-- 3 files changed, 120 insertions(+), 97 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (56e697d -> 980c6ea)
This is an automated email from the ASF dual-hosted git repository. blerer pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 56e697d Produce consistent tombstone to avoid digest mistmatch: add 833ba83 Merge branch 'cassandra-3.0' into cassandra-3.11 add ec2f2e6 Avoid storing the last schema change in CQLTester add 5a87472 Merge branch cassandra-2.2 into cassandra-3.0 add ec5e242 Merge branch cassandra-3.0 into cassandra-3.11 new 980c6ea Merge branch cassandra-3.11 into trunk The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: test/unit/org/apache/cassandra/cql3/CQLTester.java | 58 +++ .../cassandra/cql3/validation/entities/UFTest.java | 104 ++-- .../cql3/validation/entities/UFTypesTest.java | 21 ++-- .../validation/operations/AggregationTest.java | 109 ++--- 4 files changed, 159 insertions(+), 133 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (833ba83 -> ec5e242)
This is an automated email from the ASF dual-hosted git repository. blerer pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 833ba83 Merge branch 'cassandra-3.0' into cassandra-3.11 add ec2f2e6 Avoid storing the last schema change in CQLTester add 5a87472 Merge branch cassandra-2.2 into cassandra-3.0 add ec5e242 Merge branch cassandra-3.0 into cassandra-3.11 No new revisions were added by this update. Summary of changes: test/unit/org/apache/cassandra/cql3/CQLTester.java | 60 .../cassandra/cql3/validation/entities/UFTest.java | 98 ++- .../cql3/validation/entities/UFTypesTest.java | 20 ++-- .../validation/operations/AggregationTest.java | 108 ++--- 4 files changed, 157 insertions(+), 129 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-2.2 updated (0d9462e -> ec2f2e6)
This is an automated email from the ASF dual-hosted git repository. blerer pushed a change to branch cassandra-2.2 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 0d9462e Prepare debian changelog for 2.2.19 add ec2f2e6 Avoid storing the last schema change in CQLTester No new revisions were added by this update. Summary of changes: test/unit/org/apache/cassandra/cql3/CQLTester.java | 60 +++- .../cassandra/cql3/validation/entities/UFTest.java | 81 +++--- .../validation/operations/AggregationTest.java | 73 ++- 3 files changed, 120 insertions(+), 94 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch cassandra-3.11 into trunk
This is an automated email from the ASF dual-hosted git repository.
blerer pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 980c6ea6abe2f5138aa3bd2666344e86bde4b141
Merge: 56e697d ec5e242
Author: Benjamin Lerer
AuthorDate: Tue Nov 3 14:36:12 2020 +0100
Merge branch cassandra-3.11 into trunk
test/unit/org/apache/cassandra/cql3/CQLTester.java | 58 +++
.../cassandra/cql3/validation/entities/UFTest.java | 104 ++--
.../cql3/validation/entities/UFTypesTest.java | 21 ++--
.../validation/operations/AggregationTest.java | 109 ++---
4 files changed, 159 insertions(+), 133 deletions(-)
diff --cc test/unit/org/apache/cassandra/cql3/CQLTester.java
index 04ac289,4e320ef..d205c10
--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java
+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java
@@@ -713,9 -588,20 +711,19 @@@ public abstract class CQLTeste
return typeName;
}
+ protected String createFunctionName(String keyspace)
+ {
+ return String.format("%s.function_%02d", keyspace,
seqNumber.getAndIncrement());
+ }
+
+ protected void registerFunction(String functionName, String argTypes)
+ {
+ functions.add(functionName + '(' + argTypes + ')');
+ }
+
protected String createFunction(String keyspace, String argTypes, String
query) throws Throwable
{
- String functionName = String.format("%s.function_%02d", keyspace,
seqNumber.getAndIncrement());
+ String functionName = createFunctionName(keyspace);
-
createFunctionOverload(functionName, argTypes, query);
return functionName;
}
@@@ -728,9 -614,20 +736,19 @@@
schemaChange(fullQuery);
}
+ protected String createAggregateName(String keyspace)
+ {
+ return String.format("%s.aggregate_%02d", keyspace,
seqNumber.getAndIncrement());
+ }
+
+ protected void registerAggregate(String aggregateName, String argTypes)
+ {
+ aggregates.add(aggregateName + '(' + argTypes + ')');
+ }
+
protected String createAggregate(String keyspace, String argTypes, String
query) throws Throwable
{
- String aggregateName = String.format("%s.aggregate_%02d", keyspace,
seqNumber.getAndIncrement());
+ String aggregateName = createAggregateName(keyspace);
-
createAggregateOverload(aggregateName, argTypes, query);
return aggregateName;
}
@@@ -970,7 -795,7 +992,7 @@@
QueryOptions options =
QueryOptions.forInternalCalls(Collections.emptyList());
- lastSchemaChangeResult = statement.executeLocally(queryState,
options);
-return prepared.statement.executeInternal(queryState, options);
++return statement.executeLocally(queryState, options);
}
catch (Exception e)
{
diff --cc test/unit/org/apache/cassandra/cql3/validation/entities/UFTest.java
index d691374,cac0fd3..76ba6c1
--- a/test/unit/org/apache/cassandra/cql3/validation/entities/UFTest.java
+++ b/test/unit/org/apache/cassandra/cql3/validation/entities/UFTest.java
@@@ -36,13 -36,13 +36,14 @@@ import org.apache.cassandra.cql3.functi
import org.apache.cassandra.cql3.functions.UDFunction;
import org.apache.cassandra.db.marshal.CollectionType;
import org.apache.cassandra.exceptions.InvalidRequestException;
-import org.apache.cassandra.exceptions.SyntaxException;
import org.apache.cassandra.schema.KeyspaceMetadata;
+import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.service.ClientState;
- import org.apache.cassandra.transport.*;
+ import org.apache.cassandra.transport.Event.SchemaChange.Change;
+ import org.apache.cassandra.transport.Event.SchemaChange.Target;
import org.apache.cassandra.transport.ProtocolVersion;
import org.apache.cassandra.transport.messages.ResultMessage;
+import org.apache.cassandra.utils.ByteBufferUtil;
public class UFTest extends CQLTester
{
@@@ -83,58 -72,59 +84,59 @@@
@Test
public void testSchemaChange() throws Throwable
{
- String f = createFunction(KEYSPACE,
- "double, double",
- "CREATE OR REPLACE FUNCTION %s(state
double, val double) " +
- "RETURNS NULL ON NULL INPUT " +
- "RETURNS double " +
- "LANGUAGE javascript " +
- "AS '\"string\";';");
-
- assertLastSchemaChange(Event.SchemaChange.Change.CREATED,
Event.SchemaChange.Target.FUNCTION,
-KEYSPACE, parseFunctionName(f).name,
-"double", "double");
-
- createFunctionOverload(f,
-"double, double",
-"CREATE OR REPLACE FUNCTION %s(state int, v
[jira] [Updated] (CASSANDRA-16114) Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
[
https://issues.apache.org/jira/browse/CASSANDRA-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-16114:
---
Fix Version/s: 3.11.9
3.0.23
2.2.19
Since Version: 2.2.0
Source Control Link:
https://github.com/apache/cassandra/commit/ec2f2e687dde75b30c09e0a676bb03fd62ac0cbb
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Committed into cassandra-2.2 at ec2f2e687dde75b30c09e0a676bb03fd62ac0cbb and
merged into cassandra-3.0, cassandra-3.11 and trunk
> Fix tests CQLTester.assertLastSchemaChange causes ClassCastException
>
>
> Key: CASSANDRA-16114
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16114
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: David Capwell
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-beta, 2.2.19, 3.0.23, 3.11.9
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Build:
> https://app.circleci.com/pipelines/github/dcapwell/cassandra/494/workflows/b3765545-7b09-48dd-85ff-830c4f348329/jobs/2681
> {code}
> java.lang.ClassCastException:
> org.apache.cassandra.transport.messages.ResultMessage$Void cannot be cast to
> org.apache.cassandra.transport.messages.ResultMessage$SchemaChange
> at
> org.apache.cassandra.cql3.CQLTester.assertLastSchemaChange(CQLTester.java:916)
> at
> org.apache.cassandra.cql3.validation.entities.UFTest.testSchemaChange(UFTest.java:94)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-16201:
---
Attachment: 16201_jfr_40b3_alloc.png
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_40b3_alloc.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-16201:
---
Attachment: 16201_jfr_40b3_obj.png
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_40b3_alloc.png, 16201_jfr_40b3_obj.png,
> screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-16201:
---
Attachment: 16201_jfr_3118_alloc.png
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3118_alloc.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-16201:
---
Attachment: 16201_jfr_3118_obj.png
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png,
> 16201_jfr_40b3_alloc.png, 16201_jfr_40b3_obj.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-16201:
---
Attachment: 16201_jfr_3023_alloc.png
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3118_alloc.png,
> 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png, 16201_jfr_40b3_obj.png,
> screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-16201:
---
Attachment: 16201_jfr_3023_obj.png
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png,
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16227) Nodetool Netstats and tablestats tests
[ https://issues.apache.org/jira/browse/CASSANDRA-16227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225499#comment-17225499 ] Andres de la Peña commented on CASSANDRA-16227: --- Yes, just a few trivial fixes for warnings about typos, unnecessary casts, etc. in the already existent tablestats tests. Even though we have decided to not extend them during this ticket, it seemed like a good idea to do some clean-up of that minor stuff. The CI results mentioned above have already been dropped as too old, I'm running them again [here|https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch-test/45/] after a conflictless rebase. > Nodetool Netstats and tablestats tests > -- > > Key: CASSANDRA-16227 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16227 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > As part of the CASSANDRA-15583 effort we're adding unit tests for netstats > and tablestats -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-16236) Fix flaky testTrackMaxDeletionTime
[ https://issues.apache.org/jira/browse/CASSANDRA-16236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams reassigned CASSANDRA-16236: Assignee: Brandon Williams > Fix flaky testTrackMaxDeletionTime > -- > > Key: CASSANDRA-16236 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16236 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Normal > Fix For: 4.0-beta > > > * > testTrackMaxDeletionTime - org.apache.cassandra.io.sstable.SSTableMetadataTest > > junit.framework.AssertionFailedError: expected:<1.6038784E9> but > was:<1.60387827E9> > at > org.apache.cassandra.io.sstable.SSTableMetadataTest.testTrackMaxDeletionTime(SSTableMetadataTest.java:102) > https://app.circleci.com/pipelines/github/bereng/cassandra/160/workflows/6cb80410-b398-477c-b4c9-cc7369785869/jobs/1317 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225523#comment-17225523
]
Michael Semb Wever commented on CASSANDRA-16201:
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png,
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225523#comment-17225523
]
Michael Semb Wever edited comment on CASSANDRA-16201 at 11/3/20, 4:27 PM:
--
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png|width=1000!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png|width=1000!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png|width=1000!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
was (Author: michaelsembwever):
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png,
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in
[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225523#comment-17225523
]
Michael Semb Wever edited comment on CASSANDRA-16201 at 11/3/20, 4:28 PM:
--
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png|width=800!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png|width=800!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png|width=800!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
was (Author: michaelsembwever):
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png|width=1000!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png|width=1000!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png|width=1000!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png,
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a sha
[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225523#comment-17225523
]
Michael Semb Wever edited comment on CASSANDRA-16201 at 11/3/20, 4:29 PM:
--
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png|width=600!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png|width=600!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png|width=600!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
was (Author: michaelsembwever):
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png|width=800!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png|width=800!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png|width=800!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png,
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallo
[jira] [Comment Edited] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225523#comment-17225523
]
Michael Semb Wever edited comment on CASSANDRA-16201 at 11/3/20, 4:31 PM:
--
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.23
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png|width=600!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.9
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png|width=600!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta3
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png|width=600!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
was (Author: michaelsembwever):
Re-doing the screenshots as done in CASSANDRA-15430 for comparison.
h4. 2.1.18
Skipping these results as its based on the same C* code as in 15430, and I've
confirmed the allocation counts and object sizes proportionally match.
h4. 3.0.20
Allocations
* BatchMessage.execute - 1926411
** BatchStatement.getMutations => 1008170 == 52% ( previously 60%)
** BatchStatement.executeWithoutConditions => 692322 == 36% ( previously 30%)
!16201_jfr_3023_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3023_obj.png|width=600!
With the invocation count only a 1% decrease, the {{Object[]}} size has gone
from 29.8GB down to 18.7GB, or 37% reduction.
h4. 3.11.8
Allocations
* BatchMessage.execute - 1210873
** BatchStatement.getMutations => 396645 == 33% ( previously 62%)
** BatchStatement.executeWithoutConditions => 664650 == 55% ( previously 30%)
!16201_jfr_3118_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_3118_obj.png|width=600!
The {{Object[]}} size has gone from 116GB down to 8.14GB. With the invocation
count 40% that from CASSANDRA-15430, proportionally this is a 82% reduction.
h4. 4.0-beta2
Allocations
* BatchMessage.execute - 969189
** BatchStatement.getMutations => 410739 == 42% ( previously 70%)
** BatchStatement.executeWithoutConditions => 425337 == 44% ( previously 23%)
!16201_jfr_40b3_alloc.png|width=1000!
Sizes by object under {{BatchStatement.getMutations}}
!16201_jfr_40b3_obj.png|width=600!
The {{Object[]}} size has gone from 129GB down to 8.52GB. With the invocation
count 30% that from CASSANDRA-15430, proportionally this is a 78% reduction.
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png,
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallo
[jira] [Updated] (CASSANDRA-16186) SEPExecutor does not release blocked threads as it should
[
https://issues.apache.org/jira/browse/CASSANDRA-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andres de la Peña updated CASSANDRA-16186:
--
Reviewers: Andres de la Peña
> SEPExecutor does not release blocked threads as it should
> -
>
> Key: CASSANDRA-16186
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16186
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/unit
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> While adding some tests for the {{ThreadPoolMetrics}}, I discovered that the
> {{SEPExecutor}} does not release the blocked threads as it should.
> If the number of tasks added to a SEPExecutor exceed the max queue size. The
> threads adding those task will be block until enough space become available
> for all the blocked tasks. At this point all the blocked threads will
> released at once.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16201) Reduce amount of allocations during batch statement execution
[
https://issues.apache.org/jira/browse/CASSANDRA-16201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225543#comment-17225543
]
Michael Semb Wever commented on CASSANDRA-16201:
Similar to the metrics shown above from [~tsteinmaurer], 3.11 and 4.0 patches
show the most improvement. From my understanding of 15430 and this patch, this
makes sense.
[~marcuse], can you check these^ numbers and see if they align with your
expectations.
> Reduce amount of allocations during batch statement execution
> -
>
> Key: CASSANDRA-16201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16201
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Other
>Reporter: Thomas Steinmaurer
>Assignee: Marcus Eriksson
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta
>
> Attachments: 16201_jfr_3023_alloc.png, 16201_jfr_3023_obj.png,
> 16201_jfr_3118_alloc.png, 16201_jfr_3118_obj.png, 16201_jfr_40b3_alloc.png,
> 16201_jfr_40b3_obj.png, screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile,
> we see 4.0b2 going OOM from time to time. According to a heap dump, we have
> multiple NTR threads in a 3-digit MB range.
> This is likely related to object array pre-allocations at the size of
> {{BatchUpdatesCollector.updatedRows}} per {{BTree}} although there is always
> only 1 {{BTreeRow}} in the {{BTree}}.
> !screenshot-1.png|width=100%!
> So it seems we have many, many 20K elemnts pre-allocated object arrays
> resulting in a shallow heap of 80K each, although there is only one element
> in the array.
> This sort of pre-allocation is causing a lot of memory pressure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Add netstats and tablestats unit tests
This is an automated email from the ASF dual-hosted git repository.
adelapena pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 93c2d76 Add netstats and tablestats unit tests
93c2d76 is described below
commit 93c2d763eb5107c2001384700be4b240ecf7a4b8
Author: Bereng
AuthorDate: Tue Nov 3 18:06:17 2020 +
Add netstats and tablestats unit tests
patch by Berenguer Blasi; reviewed by Andrés de la Peña for CASSANDRA-16227
---
.../apache/cassandra/tools/nodetool/NetStats.java | 106 +
.../tools/nodetool/stats/StatsTableComparator.java | 11 +-
.../tools/nodetool/stats/TableStatsPrinter.java| 12 +-
.../cassandra/tools/NodetoolNetStatsTest.java | 168 +++
.../nodetool/stats/NodetoolTableStatsTest.java | 240 +
.../nodetool/stats/StatsTableComparatorTest.java | 28 ++-
.../nodetool/stats/TableStatsPrinterTest.java | 82 +++
.../tools/nodetool/stats/TableStatsTestBase.java | 117 +-
8 files changed, 583 insertions(+), 181 deletions(-)
diff --git a/src/java/org/apache/cassandra/tools/nodetool/NetStats.java
b/src/java/org/apache/cassandra/tools/nodetool/NetStats.java
index 45af43d..cc6b7b1 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/NetStats.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/NetStats.java
@@ -23,6 +23,8 @@ import io.airlift.airline.Option;
import java.io.PrintStream;
import java.util.Set;
+import com.google.common.annotations.VisibleForTesting;
+
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.net.MessagingServiceMBean;
import org.apache.cassandra.streaming.ProgressInfo;
@@ -61,63 +63,11 @@ public class NetStats extends NodeToolCmd
out.printf("%n");
if (!info.receivingSummaries.isEmpty())
{
-long totalFilesToReceive = info.getTotalFilesToReceive();
-long totalBytesToReceive = info.getTotalSizeToReceive();
-long totalFilesReceived = info.getTotalFilesReceived();
-long totalSizeReceived = info.getTotalSizeReceived();
-double percentageFilesReceived = ((double)
totalFilesReceived / totalFilesToReceive) * 100;
-double percentageSizesReceived = ((double)
totalSizeReceived / totalBytesToReceive) * 100;
-
-if (humanReadable)
-out.printf("Receiving %d files, %s total.
Already received %d files (%.2f%%), %s total (%.2f%%)%n",
- totalFilesToReceive,
-
FileUtils.stringifyFileSize(totalBytesToReceive),
- totalFilesReceived,
- percentageFilesReceived,
-
FileUtils.stringifyFileSize(totalSizeReceived),
- percentageSizesReceived);
-else
-out.printf("Receiving %d files, %d bytes
total. Already received %d files (%.2f%%), %d bytes total (%.2f%%)%n",
- totalFilesToReceive,
- totalBytesToReceive,
- totalFilesReceived,
- percentageFilesReceived,
- totalSizeReceived,
- percentageSizesReceived);
-for (ProgressInfo progress : info.getReceivingFiles())
-{
-out.printf("%s%n",
progress.toString(printPort));
-}
+printReceivingSummaries(out, info, humanReadable);
}
if (!info.sendingSummaries.isEmpty())
{
-long totalFilesToSend = info.getTotalFilesToSend();
-long totalSizeToSend = info.getTotalSizeToSend();
-long totalFilesSent = info.getTotalFilesSent();
-long totalSizeSent = info.getTotalSizeSent();
-double percentageFilesSent = ((double) totalFilesSent /
totalFilesToSend) * 100;
-double percentageSizeSent = ((double) totalSizeSent /
totalSizeToSend) * 100;
-
-if (humanReadable)
-out.printf("Sending %d files, %s total.
Already sent %d files (%.2f%%), %s total (%.2f%%)%n",
- totalFilesToSend,
-
FileUtils.stringifyFileSize(totalSizeToSend),
- totalFilesSent,
- percentageFilesSent,
- FileUtils.stringifyFileSize(totalSizeSent),
- percentageSize
[jira] [Updated] (CASSANDRA-16227) Nodetool Netstats and tablestats tests
[ https://issues.apache.org/jira/browse/CASSANDRA-16227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andres de la Peña updated CASSANDRA-16227: -- Source Control Link: https://github.com/apache/cassandra/commit/93c2d763eb5107c2001384700be4b240ecf7a4b8 Resolution: Fixed Status: Resolved (was: Ready to Commit) > Nodetool Netstats and tablestats tests > -- > > Key: CASSANDRA-16227 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16227 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > As part of the CASSANDRA-15583 effort we're adding unit tests for netstats > and tablestats -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16227) Nodetool Netstats and tablestats tests
[ https://issues.apache.org/jira/browse/CASSANDRA-16227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225592#comment-17225592 ] Andres de la Peña commented on CASSANDRA-16227: --- Committed to trunk as [93c2d763eb5107c2001384700be4b240ecf7a4b8|https://github.com/apache/cassandra/commit/93c2d763eb5107c2001384700be4b240ecf7a4b8]. > Nodetool Netstats and tablestats tests > -- > > Key: CASSANDRA-16227 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16227 > Project: Cassandra > Issue Type: Improvement > Components: Test/unit >Reporter: Berenguer Blasi >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > As part of the CASSANDRA-15583 effort we're adding unit tests for netstats > and tablestats -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16146) Node state incorrectly set to NORMAL after nodetool disablegossip and enablegossip during bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225750#comment-17225750
]
Yifan Cai commented on CASSANDRA-16146:
---
Fixups and CI
3.0: https://github.com/yifan-c/cassandra/tree/fixup/CASSANDRA-16146-3.0
https://app.circleci.com/pipelines/github/yifan-c/cassandra/143/workflows/1a55cee6-36ef-4e8a-a8d4-6798549ca202
3.11: https://github.com/yifan-c/cassandra/tree/fixup/CASSANDRA-16146-3.11
https://app.circleci.com/pipelines/github/yifan-c/cassandra/144/workflows/52cfbd53-92cb-4644-ae0d-694a9bef398c
trunk: https://github.com/yifan-c/cassandra/tree/fixup/CASSANDRA-16146-trunk
https://app.circleci.com/pipelines/github/yifan-c/cassandra/142/workflows/480e513f-2cf5-47b7-a608-d37cb91e1393
> Node state incorrectly set to NORMAL after nodetool disablegossip and
> enablegossip during bootstrap
> ---
>
> Key: CASSANDRA-16146
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16146
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Gossip
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.0-beta3
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> At high level, {{StorageService#setGossipTokens}} set the gossip state to
> {{NORMAL}} blindly. Therefore, re-enabling gossip (stop and start gossip)
> overrides the actual gossip state.
>
> It could happen in the below scenario.
> # Bootstrap failed. The gossip state remains in {{BOOT}} / {{JOINING}} and
> code execution exits StorageService#initServer.
> # Operator runs nodetool to stop and re-start gossip. The gossip state gets
> flipped to {{NORMAL}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16083) Missing JMX objects and attributes upgrading from 3.0 to 4.0
[
https://issues.apache.org/jira/browse/CASSANDRA-16083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225826#comment-17225826
]
Ekaterina Dimitrova commented on CASSANDRA-16083:
-
Short update:
Objects:
*"org.apache.cassandra.metrics:type=ThreadPools.*"* ---> a few scopes are
missing which I still haven't found how that happened as the stages that formed
the scope look the same but some refactoring of the code happened so I need to
check actual details.
*"org.apache.cassandra.metrics:type=DroppedMessage.*" *---> as part of
CASSANDRA-15066 verbs have been changed. Many are now seen split to Verb_RSP
and Verb_REQ, for example MUTATION_RSP and MUTATION_RSQ.
*"org.apache.cassandra.internal:.*" *- I didn't really dig into it but looks
like a few types are missing now in 4.0 - AntiEntropyStage,
InternalResponseStage, MiscStage, PerDiskMemtableFlushWriter_0,
ViewBuildExecutor
*"org.apache.cassandra.metrics:type=ClientRequest,scope=CASRead,name=ConditionNotMet"*
---> conditionNotMet has been moved to CASClientWriteRequestMetrics as part of
CASSANDRA-12649. What is interesting to me is that in 3.11 we see in
CASClientRequestMetrics:
{code:java}
/* Used only for write */
public final Counter conditionNotMet; in 3.11
{code}
> Missing JMX objects and attributes upgrading from 3.0 to 4.0
>
>
> Key: CASSANDRA-16083
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16083
> Project: Cassandra
> Issue Type: Task
> Components: Observability/Metrics
>Reporter: David Capwell
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Using the tools added in CASSANDRA-16082, below are the list of metrics
> missing in 4.0 but present in 3.0. The work here is to make sure we had
> proper deprecation for each metric, and if not to add it back.
> {code}
> $ tools/bin/jmxtool diff -f yaml cassandra-3.0-jmx.yaml trunk-jmx.yaml
> --ignore-missing-on-left
> Objects not in right:
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=schema_columnfamilies,name=CasPrepareLatency
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=batchlog,name=EstimatedPartitionSizeHistogram
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=hints,name=BloomFilterFalseRatio
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=views_builds_in_progress,name=ReplicaFilteringProtectionRequests
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=batchlog,name=RowCacheHitOutOfRange
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=views_builds_in_progress,name=CasPrepareLatency
> org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=CounterMutationStage,name=MaxPoolSize
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=views_builds_in_progress,name=ColUpdateTimeDeltaHistogram
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=batchlog,name=TombstoneScannedHistogram
> org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=ReadStage,name=ActiveTasks
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=hints,name=WaitingOnFreeMemtableSpace
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=schema_columnfamilies,name=CasCommitTotalLatency
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=hints,name=MemtableOnHeapSize
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=schema_aggregates,name=CasProposeLatency
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=batchlog,name=AllMemtablesLiveDataSize
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=hints,name=ViewReadTime
> org.apache.cassandra.db:type=HintedHandoffManager
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=batchlog,name=BloomFilterDiskSpaceUsed
> org.apache.cassandra.metrics:type=ThreadPools,path=request,scope=RequestResponseStage,name=PendingTasks
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=views_builds_in_progress,name=MemtableSwitchCount
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=hints,name=MemtableOnHeapSize
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=range_xfers,name=ReplicaFilteringProtectionRowsCachedPerQuery
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=views_builds_in_progress,name=SnapshotsSize
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=batchlog,name=RecentBloomFilterFalsePositives
> org.apache.cassandra.metrics:type=Table,keyspace=system,scope=views_builds_in_progress,name=ColUpdateTimeDeltaHistogram
> org.apache.cassandra.metrics:type=ColumnFamily,keyspace=system,scope=range_xfers,name=SpeculativeRetries
> org.ap
[jira] [Commented] (CASSANDRA-16066) Update and rework cassandra-website material to work with Antora
[ https://issues.apache.org/jira/browse/CASSANDRA-16066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225840#comment-17225840 ] Anthony Grasso commented on CASSANDRA-16066: I've started prototyping what tooling would look like for the cassandra-website repository if it used Anotra. What I've done so far can be found on [my fork|https://github.com/ossarga/cassandra-website] of the cassandra-website on a branch called [anthony/CASSANDRA-16066|https://github.com/ossarga/cassandra-website/tree/anthony/CASSANDRA-16066]. At the time of writing it addresses the following points in the previous comment: * Tooling can generate entire website including in-tree docs * Address legalities of conflicting licenses (unable to store Antora's UI bundle and generation in-tree). * Generated content is never committed to the master branch of the cassandra-website (relevant to Cassandra releases). * Continue to maintain staging branch and live branch in cassandra-website. Todo items: * Push to cassandra-website staging branch. * Preview mode for (in-tree) docs when editing. * Preview mode for when editing UI components. * Cassandra website CI only tiggers if there are changes in the website repository or when Cassandra in-tree docs change (can be made to listen changes in multiple repositories). * Ability to generate in-tree docs without using Antora so it can be bundled in with a Cassandra release (Simple AsciiDoc build as a starting point). > Update and rework cassandra-website material to work with Antora > > > Key: CASSANDRA-16066 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16066 > Project: Cassandra > Issue Type: Task > Components: Documentation/Website >Reporter: Anthony Grasso >Priority: Normal > Attachments: image-2020-09-05-13-24-13-606.png, > image-2020-09-06-07-17-14-705.png > > > *We want to modernise the way the website is built* > Rework the cassandra-website repository to generate a UI bundle containing > resources that Antora will use when generating the Cassandra documents and > website. > *The existing method is starting to become dated* > The documentation and website templates are currently in markdown format. > Sphinx is used to generate the Cassandra documentation and Jekyll generates > the website content. One of the major issues with the existing website > tooling is that the live preview server (render site as it is being updated) > is really slow. There is a preview server that is really fast, however it is > unable to detect changes to the content and render automatically. > *We are migrating the docs to be rendered with Antora* > The work in CASSANDRA-16029 is converting the document templates to AsciiDoc > format. Sphinx is being replaced by Antora to generate the documentation > content. This change has two advantages: > * More flexibility if the Apache Cassandra documentation look and feel needs > to be updated or redesigned. > * More modern look and feel to the documentation. > *We can use Antora to generate the website as well* > Antora could also be used to generate the Cassandra website content. As > suggested on the [mailing > list|https://www.mail-archive.com/dev@cassandra.apache.org/msg15577.html] > this would require the existing markdown templates to be converted to > AsciiDoc as well. > *Antora needs a UI bundle to style content* > For Antora to generate the document content and potentially the website > content it requires a UI bundle (ui-bundle.zip). The UI bundle contains the > HTML templates (layouts, partials, and helpers), CSS, JavaScript, fonts, and > (site-wide) images. As such, it provides both the visual theme and user > interactions for the documentation. Effectively the UI bundle is the > templates and styling that are applied to the documentation and website > content. > *The [cassandra-website|https://github.com/apache/cassandra-website] > repository can be used to generate the UI bundle* > All the resources associated with templating and styling the documentation > and website can be placed in the > [cassandra-website|https://github.com/apache/cassandra-website] repository. > In this case the repository would serve two purposes; > * Generation of the UI bundle resources. > * Serve the production website content. > *The [cassandra|https://github.com/apache/cassandra] repository would contain > the documentation material, while the rest of the non-versioned pages would > contain that material* > * All other material that is used to generate documentation would live in > the [cassandra|https://github.com/apache/cassandra] repository. In this case > Antora would run on the > [cassandra/doc|https://github.com/apache/cassandra/doc] directory and pull in
[jira] [Comment Edited] (CASSANDRA-16066) Update and rework cassandra-website material to work with Antora
[ https://issues.apache.org/jira/browse/CASSANDRA-16066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225840#comment-17225840 ] Anthony Grasso edited comment on CASSANDRA-16066 at 11/4/20, 4:25 AM: -- I've started prototyping what tooling would look like for the cassandra-website repository if it used Anotra. What I've done so far can be found on [my fork|https://github.com/ossarga/cassandra-website] of the cassandra-website on a branch called [anthony/CASSANDRA-16066|https://github.com/ossarga/cassandra-website/tree/anthony/CASSANDRA-16066]. At the time of writing it addresses the following points in the previous comment: * Tooling can generate entire website including in-tree docs * Address legalities of conflicting licenses (unable to store Antora's UI bundle and generation in-tree). * Generated content is never committed to the master branch of the cassandra-website (relevant to Cassandra releases). * Continue to maintain staging branch and live branch in cassandra-website. Todo items: * Push to cassandra-website staging branch. * Preview mode for (in-tree) docs when editing. * Preview mode for UI component editing. * Cassandra website CI only tiggers if there are changes in the website repository or when Cassandra in-tree docs change (can be made to listen changes in multiple repositories). * Ability to generate in-tree docs without using Antora so it can be bundled in with a Cassandra release (Simple AsciiDoc build as a starting point). was (Author: anthony grasso): I've started prototyping what tooling would look like for the cassandra-website repository if it used Anotra. What I've done so far can be found on [my fork|https://github.com/ossarga/cassandra-website] of the cassandra-website on a branch called [anthony/CASSANDRA-16066|https://github.com/ossarga/cassandra-website/tree/anthony/CASSANDRA-16066]. At the time of writing it addresses the following points in the previous comment: * Tooling can generate entire website including in-tree docs * Address legalities of conflicting licenses (unable to store Antora's UI bundle and generation in-tree). * Generated content is never committed to the master branch of the cassandra-website (relevant to Cassandra releases). * Continue to maintain staging branch and live branch in cassandra-website. Todo items: * Push to cassandra-website staging branch. * Preview mode for (in-tree) docs when editing. * Preview mode for when editing UI components. * Cassandra website CI only tiggers if there are changes in the website repository or when Cassandra in-tree docs change (can be made to listen changes in multiple repositories). * Ability to generate in-tree docs without using Antora so it can be bundled in with a Cassandra release (Simple AsciiDoc build as a starting point). > Update and rework cassandra-website material to work with Antora > > > Key: CASSANDRA-16066 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16066 > Project: Cassandra > Issue Type: Task > Components: Documentation/Website >Reporter: Anthony Grasso >Priority: Normal > Attachments: image-2020-09-05-13-24-13-606.png, > image-2020-09-06-07-17-14-705.png > > > *We want to modernise the way the website is built* > Rework the cassandra-website repository to generate a UI bundle containing > resources that Antora will use when generating the Cassandra documents and > website. > *The existing method is starting to become dated* > The documentation and website templates are currently in markdown format. > Sphinx is used to generate the Cassandra documentation and Jekyll generates > the website content. One of the major issues with the existing website > tooling is that the live preview server (render site as it is being updated) > is really slow. There is a preview server that is really fast, however it is > unable to detect changes to the content and render automatically. > *We are migrating the docs to be rendered with Antora* > The work in CASSANDRA-16029 is converting the document templates to AsciiDoc > format. Sphinx is being replaced by Antora to generate the documentation > content. This change has two advantages: > * More flexibility if the Apache Cassandra documentation look and feel needs > to be updated or redesigned. > * More modern look and feel to the documentation. > *We can use Antora to generate the website as well* > Antora could also be used to generate the Cassandra website content. As > suggested on the [mailing > list|https://www.mail-archive.com/dev@cassandra.apache.org/msg15577.html] > this would require the existing markdown templates to be converted to > AsciiDoc as well. > *Antora needs a UI bundle to style content* > For A
[jira] [Commented] (CASSANDRA-16213) Cannot replace_address /X because it doesn't exist in gossip
[ https://issues.apache.org/jira/browse/CASSANDRA-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17225842#comment-17225842 ] Ekaterina Dimitrova commented on CASSANDRA-16213: - I left a few small comments/questions on the CASSANDRA_RELEVANT_PROPERTIES' new methods, as per request. I see you already have two reviewers committers but let me know if you want me to do a review of the full ticket. > Cannot replace_address /X because it doesn't exist in gossip > > > Key: CASSANDRA-16213 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16213 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip, Cluster/Membership >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-beta > > > We see this exception around nodes crashing and trying to do a host > replacement; this error appears to be correlated around multiple node > failures. > A simplified case to trigger this is the following > *) Have a N node cluster > *) Shutdown all N nodes > *) Bring up N-1 nodes (at least 1 seed, else replace seed) > *) Host replace the N-1th node -> this will fail with the above > The reason this happens is that the N-1th node isn’t gossiping anymore, and > the existing nodes do not have its details in gossip (but have the details in > the peers table), so the host replacement fails as the node isn’t known in > gossip. > This affects all versions (tested 3.0 and trunk, assume 2.2 as well) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
