[jira] [Comment Edited] (CASSANDRA-9563) Rename class for DATE type in Java driver
[ https://issues.apache.org/jira/browse/CASSANDRA-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591349#comment-14591349 ] Robert Stupp edited comment on CASSANDRA-9563 at 6/18/15 6:42 AM: -- tests look good (i.e. no regressions): [2.2 testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-testall/1/] [2.2 dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-dtest/1/] [trunk testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-trunk-testall/1/] [trunk dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-trunk-dtest/1/] was (Author: snazy): tests look good (i.e. no regressions): [2.2 testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-testall/1/| [2.2 dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-dtest/1/| [trunk testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-trunk-testall/1/| [trunk dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-9563-rename-DateWithoutTime-trunk-dtest/1/| > Rename class for DATE type in Java driver > - > > Key: CASSANDRA-9563 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9563 > Project: Cassandra > Issue Type: Improvement >Reporter: Olivier Michallat >Assignee: Robert Stupp >Priority: Minor > Fix For: 2.2.0 rc2 > > > An early preview of the Java driver 2.2 was provided for inclusion in > Cassandra 2.2.0-rc1. It uses a custom Java type to represent CQL type > {{DATE}}. Currently that Java type is called {{DateWithoutTime}}. > We'd like to rename it to {{LocalDate}}. This would be a breaking change for > Cassandra, because that type is visible from UDF implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9594) metrics reporter doesn't start until after a bootstrap
[ https://issues.apache.org/jira/browse/CASSANDRA-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591320#comment-14591320 ] Tommy Stendahl commented on CASSANDRA-9594: --- I did some checking in the code and it looks perfectly safe to start the metrics reporter just before the call to {{o.a.c.service.StorageService#registerDaemon}}, or do I miss some side effects here? > metrics reporter doesn't start until after a bootstrap > -- > > Key: CASSANDRA-9594 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9594 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eric Evans >Priority: Minor > Labels: lhf > > In {{o.a.c.service.CassandraDaemon#setup}}, the metrics reporter is started > immediately after the invocation of > {{o.a.c.service.StorageService#initServer}}, which for a bootstrapping node > may block for a considerable period of time. If the metrics reporter is your > only source of visibility, then you are blind until the bootstrap completes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[6/6] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/43d21c38 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/43d21c38 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/43d21c38 Branch: refs/heads/trunk Commit: 43d21c384b4212f8731edfae33142c0a5676c474 Parents: 4c4c432 dee675f Author: Yuki Morishita Authored: Wed Jun 17 21:58:02 2015 -0500 Committer: Yuki Morishita Committed: Wed Jun 17 21:58:02 2015 -0500 -- CHANGES.txt | 3 +- .../cassandra/service/StorageService.java | 48 +--- 2 files changed, 34 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/43d21c38/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/43d21c38/src/java/org/apache/cassandra/service/StorageService.java --
[5/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dee675f1 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dee675f1 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dee675f1 Branch: refs/heads/cassandra-2.2 Commit: dee675f1ef148b40351c365b6d42c39f081cb706 Parents: 8b021db 9966419 Author: Yuki Morishita Authored: Wed Jun 17 20:48:40 2015 -0500 Committer: Yuki Morishita Committed: Wed Jun 17 20:48:40 2015 -0500 -- CHANGES.txt | 3 +- .../cassandra/service/StorageService.java | 48 +--- 2 files changed, 34 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/dee675f1/CHANGES.txt -- diff --cc CHANGES.txt index c32596c,1d72c9a..3b16b6f --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,41 -1,13 +1,42 @@@ -2.1.7 +2.2 + * Fix connection leak in CqlRecordWriter (CASSANDRA-9576) + * Mlockall before opening system sstables & remove boot_without_jna option (CASSANDRA-9573) + * Add functions to convert timeuuid to date or time, deprecate dateOf and unixTimestampOf (CASSANDRA-9229) + * Make sure we cancel non-compacting sstables from LifecycleTransaction (CASSANDRA-9566) + * Fix deprecated repair JMX API (CASSANDRA-9570) ++ * Add logback metrics (CASSANDRA-9378) +Merged from 2.1: * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) ++ * Make rebuild only run one at a time (CASSANDRA-9119) Merged from 2.0 * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) - * Add logback metrics (CASSANDRA-9378) - * Make rebuild only run one at a time (CASSANDRA-9119) -2.1.6 +2.2.0-rc1 + * Compressed commit log should measure compressed space used (CASSANDRA-9095) + * Fix comparison bug in CassandraRoleManager#collectRoles (CASSANDRA-9551) + * Add tinyint,smallint,time,date support for UDFs (CASSANDRA-9400) + * Deprecates SSTableSimpleWriter and SSTableSimpleUnsortedWriter (CASSANDRA-9546) + * Empty INITCOND treated as null in aggregate (CASSANDRA-9457) + * Remove use of Cell in Thrift MapReduce classes (CASSANDRA-8609) + * Integrate pre-release Java Driver 2.2-rc1, custom build (CASSANDRA-9493) + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) + * ant test-all results incomplete when parsed (CASSANDRA-9463) + * Disallow frozen<> types in function arguments and return types for + clarity (CASSANDRA-9411) + * Static Analysis to warn on unsafe use of Autocloseable instances (CASSANDRA-9431) + * Update commitlog archiving examples now that commitlog segments are + not recycled (CASSANDRA-9350) + * Extend Transactional API to sstable lifecycle management (CASSANDRA-8568) + * (cqlsh) Add support for native protocol 4 (CASSANDRA-9399) + * Ensure that UDF and UDAs are keyspace-isolated (CASSANDRA-9409) + * Revert CASSANDRA-7807 (tracing completion client notifications) (CASSANDRA-9429) + * Add ability to stop compaction by ID (CASSANDRA-7207) + * Let CassandraVersion handle SNAPSHOT version (CASSANDRA-9438) +Merged from 2.1: * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) * Use ProtocolError code instead of ServerError code for native protocol http://git-wip-us.apache.org/repos/asf/cassandra/blob/dee675f1/src/java/org/apache/cassandra/service/StorageService.java -- diff --cc src/java/org/apache/cassandra/service/StorageService.java index 2dd56b5,e063c63..3edbe22 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@@ -50,7 -50,9 +50,8 @@@ import java.util.concurrent.Future import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; + import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; -import java.util.concurrent.atomic.AtomicLong; import javax.management.JMX; import javax.management.MBeanServer; @@@ -239,11 -235,15 +240,13 @@@ public class StorageService extends Not private InetAddress removingNode; /* Are we starting this node in bootstrap mode? */ -private boolean isBootstrapMode; +private volatile boolean isBootstrapMode; /* we bootstrap but do NOT join the ring unless to
[2/6] cassandra git commit: Make rebuild only run one at a time
Make rebuild only run one at a time patch by yukim; reviewed by jmckenzie for CASSANDRA-9119 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9966419d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9966419d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9966419d Branch: refs/heads/cassandra-2.2 Commit: 9966419dbda995421f41ccc769d3b89d63940c82 Parents: 4c15970 Author: Yuki Morishita Authored: Wed Jun 3 14:44:11 2015 -0500 Committer: Yuki Morishita Committed: Wed Jun 17 20:41:16 2015 -0500 -- CHANGES.txt | 1 + .../cassandra/service/StorageService.java | 36 ++-- 2 files changed, 27 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9966419d/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 8f3f9f0..1d72c9a 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -4,6 +4,7 @@ Merged from 2.0 * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) + * Make rebuild only run one at a time (CASSANDRA-9119) 2.1.6 http://git-wip-us.apache.org/repos/asf/cassandra/blob/9966419d/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 7c8e424..e063c63 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -50,6 +50,7 @@ import java.util.concurrent.Future; import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; @@ -237,7 +238,9 @@ public class StorageService extends NotificationBroadcasterSupport implements IE private boolean isBootstrapMode; /* we bootstrap but do NOT join the ring unless told to do so */ -private boolean isSurveyMode= Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false")); +private boolean isSurveyMode = Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false")); +/* true if node is rebuilding and receiving data */ +private final AtomicBoolean isRebuilding = new AtomicBoolean(); /* when intialized as a client, we shouldn't write to the system keyspace. */ private boolean isClientMode; @@ -1023,19 +1026,27 @@ public class StorageService extends NotificationBroadcasterSupport implements IE public void rebuild(String sourceDc) { -logger.info("rebuild from dc: {}", sourceDc == null ? "(any dc)" : sourceDc); - -RangeStreamer streamer = new RangeStreamer(tokenMetadata, FBUtilities.getBroadcastAddress(), "Rebuild"); -streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance)); -if (sourceDc != null) -streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc)); +// check on going rebuild +if (!isRebuilding.compareAndSet(false, true)) +{ +throw new IllegalStateException("Node is still rebuilding. Check nodetool netstats."); +} -for (String keyspaceName : Schema.instance.getNonSystemKeyspaces()) -streamer.addRanges(keyspaceName, getLocalRanges(keyspaceName)); +logger.info("rebuild from dc: {}", sourceDc == null ? "(any dc)" : sourceDc); try { -streamer.fetchAsync().get(); +RangeStreamer streamer = new RangeStreamer(tokenMetadata, FBUtilities.getBroadcastAddress(), "Rebuild"); +streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance)); +if (sourceDc != null) +streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc)); + +for (String keyspaceName : Schema.instance.getNonSystemKeyspaces()) +streamer.addRanges(keyspaceName, getLocalRanges(keyspaceName)); + +StreamResultFuture resultFuture = streamer.fetchAsync(); +// wait for result +resultFuture.get(); } catch (InterruptedException e) { @@ -1047,6 +1058,11 @@ public class StorageService extends NotificationBroadcaster
[1/6] cassandra git commit: Make rebuild only run one at a time
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 4c1597011 -> 9966419db refs/heads/cassandra-2.2 8b021db7c -> dee675f1e refs/heads/trunk 4c4c4327a -> 43d21c384 Make rebuild only run one at a time patch by yukim; reviewed by jmckenzie for CASSANDRA-9119 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9966419d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9966419d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9966419d Branch: refs/heads/cassandra-2.1 Commit: 9966419dbda995421f41ccc769d3b89d63940c82 Parents: 4c15970 Author: Yuki Morishita Authored: Wed Jun 3 14:44:11 2015 -0500 Committer: Yuki Morishita Committed: Wed Jun 17 20:41:16 2015 -0500 -- CHANGES.txt | 1 + .../cassandra/service/StorageService.java | 36 ++-- 2 files changed, 27 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9966419d/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 8f3f9f0..1d72c9a 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -4,6 +4,7 @@ Merged from 2.0 * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) + * Make rebuild only run one at a time (CASSANDRA-9119) 2.1.6 http://git-wip-us.apache.org/repos/asf/cassandra/blob/9966419d/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 7c8e424..e063c63 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -50,6 +50,7 @@ import java.util.concurrent.Future; import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; @@ -237,7 +238,9 @@ public class StorageService extends NotificationBroadcasterSupport implements IE private boolean isBootstrapMode; /* we bootstrap but do NOT join the ring unless told to do so */ -private boolean isSurveyMode= Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false")); +private boolean isSurveyMode = Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false")); +/* true if node is rebuilding and receiving data */ +private final AtomicBoolean isRebuilding = new AtomicBoolean(); /* when intialized as a client, we shouldn't write to the system keyspace. */ private boolean isClientMode; @@ -1023,19 +1026,27 @@ public class StorageService extends NotificationBroadcasterSupport implements IE public void rebuild(String sourceDc) { -logger.info("rebuild from dc: {}", sourceDc == null ? "(any dc)" : sourceDc); - -RangeStreamer streamer = new RangeStreamer(tokenMetadata, FBUtilities.getBroadcastAddress(), "Rebuild"); -streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance)); -if (sourceDc != null) -streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc)); +// check on going rebuild +if (!isRebuilding.compareAndSet(false, true)) +{ +throw new IllegalStateException("Node is still rebuilding. Check nodetool netstats."); +} -for (String keyspaceName : Schema.instance.getNonSystemKeyspaces()) -streamer.addRanges(keyspaceName, getLocalRanges(keyspaceName)); +logger.info("rebuild from dc: {}", sourceDc == null ? "(any dc)" : sourceDc); try { -streamer.fetchAsync().get(); +RangeStreamer streamer = new RangeStreamer(tokenMetadata, FBUtilities.getBroadcastAddress(), "Rebuild"); +streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance)); +if (sourceDc != null) +streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc)); + +for (String keyspaceName : Schema.instance.getNonSystemKeyspaces()) +streamer.addRanges(keyspaceName, getLocalRanges(keyspaceName)); + +StreamResultFuture resultFuture = streamer.fetchAsync(); +// wait for r
[4/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dee675f1 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dee675f1 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dee675f1 Branch: refs/heads/trunk Commit: dee675f1ef148b40351c365b6d42c39f081cb706 Parents: 8b021db 9966419 Author: Yuki Morishita Authored: Wed Jun 17 20:48:40 2015 -0500 Committer: Yuki Morishita Committed: Wed Jun 17 20:48:40 2015 -0500 -- CHANGES.txt | 3 +- .../cassandra/service/StorageService.java | 48 +--- 2 files changed, 34 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/dee675f1/CHANGES.txt -- diff --cc CHANGES.txt index c32596c,1d72c9a..3b16b6f --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,41 -1,13 +1,42 @@@ -2.1.7 +2.2 + * Fix connection leak in CqlRecordWriter (CASSANDRA-9576) + * Mlockall before opening system sstables & remove boot_without_jna option (CASSANDRA-9573) + * Add functions to convert timeuuid to date or time, deprecate dateOf and unixTimestampOf (CASSANDRA-9229) + * Make sure we cancel non-compacting sstables from LifecycleTransaction (CASSANDRA-9566) + * Fix deprecated repair JMX API (CASSANDRA-9570) ++ * Add logback metrics (CASSANDRA-9378) +Merged from 2.1: * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) ++ * Make rebuild only run one at a time (CASSANDRA-9119) Merged from 2.0 * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) - * Add logback metrics (CASSANDRA-9378) - * Make rebuild only run one at a time (CASSANDRA-9119) -2.1.6 +2.2.0-rc1 + * Compressed commit log should measure compressed space used (CASSANDRA-9095) + * Fix comparison bug in CassandraRoleManager#collectRoles (CASSANDRA-9551) + * Add tinyint,smallint,time,date support for UDFs (CASSANDRA-9400) + * Deprecates SSTableSimpleWriter and SSTableSimpleUnsortedWriter (CASSANDRA-9546) + * Empty INITCOND treated as null in aggregate (CASSANDRA-9457) + * Remove use of Cell in Thrift MapReduce classes (CASSANDRA-8609) + * Integrate pre-release Java Driver 2.2-rc1, custom build (CASSANDRA-9493) + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) + * ant test-all results incomplete when parsed (CASSANDRA-9463) + * Disallow frozen<> types in function arguments and return types for + clarity (CASSANDRA-9411) + * Static Analysis to warn on unsafe use of Autocloseable instances (CASSANDRA-9431) + * Update commitlog archiving examples now that commitlog segments are + not recycled (CASSANDRA-9350) + * Extend Transactional API to sstable lifecycle management (CASSANDRA-8568) + * (cqlsh) Add support for native protocol 4 (CASSANDRA-9399) + * Ensure that UDF and UDAs are keyspace-isolated (CASSANDRA-9409) + * Revert CASSANDRA-7807 (tracing completion client notifications) (CASSANDRA-9429) + * Add ability to stop compaction by ID (CASSANDRA-7207) + * Let CassandraVersion handle SNAPSHOT version (CASSANDRA-9438) +Merged from 2.1: * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) * Use ProtocolError code instead of ServerError code for native protocol http://git-wip-us.apache.org/repos/asf/cassandra/blob/dee675f1/src/java/org/apache/cassandra/service/StorageService.java -- diff --cc src/java/org/apache/cassandra/service/StorageService.java index 2dd56b5,e063c63..3edbe22 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@@ -50,7 -50,9 +50,8 @@@ import java.util.concurrent.Future import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; + import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; -import java.util.concurrent.atomic.AtomicLong; import javax.management.JMX; import javax.management.MBeanServer; @@@ -239,11 -235,15 +240,13 @@@ public class StorageService extends Not private InetAddress removingNode; /* Are we starting this node in bootstrap mode? */ -private boolean isBootstrapMode; +private volatile boolean isBootstrapMode; /* we bootstrap but do NOT join the ring unless told to do
[3/6] cassandra git commit: Make rebuild only run one at a time
Make rebuild only run one at a time patch by yukim; reviewed by jmckenzie for CASSANDRA-9119 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9966419d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9966419d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9966419d Branch: refs/heads/trunk Commit: 9966419dbda995421f41ccc769d3b89d63940c82 Parents: 4c15970 Author: Yuki Morishita Authored: Wed Jun 3 14:44:11 2015 -0500 Committer: Yuki Morishita Committed: Wed Jun 17 20:41:16 2015 -0500 -- CHANGES.txt | 1 + .../cassandra/service/StorageService.java | 36 ++-- 2 files changed, 27 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9966419d/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 8f3f9f0..1d72c9a 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -4,6 +4,7 @@ Merged from 2.0 * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) + * Make rebuild only run one at a time (CASSANDRA-9119) 2.1.6 http://git-wip-us.apache.org/repos/asf/cassandra/blob/9966419d/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 7c8e424..e063c63 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -50,6 +50,7 @@ import java.util.concurrent.Future; import java.util.concurrent.FutureTask; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; @@ -237,7 +238,9 @@ public class StorageService extends NotificationBroadcasterSupport implements IE private boolean isBootstrapMode; /* we bootstrap but do NOT join the ring unless told to do so */ -private boolean isSurveyMode= Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false")); +private boolean isSurveyMode = Boolean.parseBoolean(System.getProperty("cassandra.write_survey", "false")); +/* true if node is rebuilding and receiving data */ +private final AtomicBoolean isRebuilding = new AtomicBoolean(); /* when intialized as a client, we shouldn't write to the system keyspace. */ private boolean isClientMode; @@ -1023,19 +1026,27 @@ public class StorageService extends NotificationBroadcasterSupport implements IE public void rebuild(String sourceDc) { -logger.info("rebuild from dc: {}", sourceDc == null ? "(any dc)" : sourceDc); - -RangeStreamer streamer = new RangeStreamer(tokenMetadata, FBUtilities.getBroadcastAddress(), "Rebuild"); -streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance)); -if (sourceDc != null) -streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc)); +// check on going rebuild +if (!isRebuilding.compareAndSet(false, true)) +{ +throw new IllegalStateException("Node is still rebuilding. Check nodetool netstats."); +} -for (String keyspaceName : Schema.instance.getNonSystemKeyspaces()) -streamer.addRanges(keyspaceName, getLocalRanges(keyspaceName)); +logger.info("rebuild from dc: {}", sourceDc == null ? "(any dc)" : sourceDc); try { -streamer.fetchAsync().get(); +RangeStreamer streamer = new RangeStreamer(tokenMetadata, FBUtilities.getBroadcastAddress(), "Rebuild"); +streamer.addSourceFilter(new RangeStreamer.FailureDetectorSourceFilter(FailureDetector.instance)); +if (sourceDc != null) +streamer.addSourceFilter(new RangeStreamer.SingleDatacenterFilter(DatabaseDescriptor.getEndpointSnitch(), sourceDc)); + +for (String keyspaceName : Schema.instance.getNonSystemKeyspaces()) +streamer.addRanges(keyspaceName, getLocalRanges(keyspaceName)); + +StreamResultFuture resultFuture = streamer.fetchAsync(); +// wait for result +resultFuture.get(); } catch (InterruptedException e) { @@ -1047,6 +1058,11 @@ public class StorageService extends NotificationBroadcasterSupport
cassandra git commit: use short-circuiting ops
Repository: cassandra Updated Branches: refs/heads/trunk 72ab4b944 -> 4c4c4327a use short-circuiting ops Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4c4c4327 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4c4c4327 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4c4c4327 Branch: refs/heads/trunk Commit: 4c4c4327a2f7653b27dd89a9a1f1ae024d7c76be Parents: 72ab4b9 Author: Dave Brosius Authored: Wed Jun 17 22:23:23 2015 -0400 Committer: Dave Brosius Committed: Wed Jun 17 22:23:23 2015 -0400 -- .../apache/cassandra/io/util/UnbufferedDataOutputStreamPlus.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c4c4327/src/java/org/apache/cassandra/io/util/UnbufferedDataOutputStreamPlus.java -- diff --git a/src/java/org/apache/cassandra/io/util/UnbufferedDataOutputStreamPlus.java b/src/java/org/apache/cassandra/io/util/UnbufferedDataOutputStreamPlus.java index 087b888..8f2cee0 100644 --- a/src/java/org/apache/cassandra/io/util/UnbufferedDataOutputStreamPlus.java +++ b/src/java/org/apache/cassandra/io/util/UnbufferedDataOutputStreamPlus.java @@ -309,7 +309,7 @@ public abstract class UnbufferedDataOutputStreamPlus extends DataOutputStreamPlu for (int i = 0 ; i < charRunLength ; i++) { char ch = str.charAt(offset + i); -if ((ch > 0) & (ch <= 127)) +if ((ch > 0) && (ch <= 127)) { utfBytes[utfIndex++] = (byte) ch; }
[jira] [Resolved] (CASSANDRA-9614) I am trying to execute queries using execute_concurrent() in python but I am getting the best throughput on a concurrency of 4 whereas according to the documentation
[ https://issues.apache.org/jira/browse/CASSANDRA-9614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-9614. --- Resolution: Invalid Fix Version/s: (was: 1.2.x) > I am trying to execute queries using execute_concurrent() in python but I am > getting the best throughput on a concurrency of 4 whereas according to the > documentation there can be 100 concurrent requests. Why this kind of behavior? > -- > > Key: CASSANDRA-9614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9614 > Project: Cassandra > Issue Type: Test > Components: Documentation & website > Environment: Linux >Reporter: Kumar Saras > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9496) ArrivalWindow should use primitives
[ https://issues.apache.org/jira/browse/CASSANDRA-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-9496: --- Fix Version/s: 2.2.0 rc2 2.1.7 2.0.16 3.x > ArrivalWindow should use primitives > > > Key: CASSANDRA-9496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9496 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Minor > Fix For: 3.x, 2.0.16, 2.1.7, 2.2.0 rc2 > > Attachments: 9496.txt > > > While doing a heap analysis of a large cluster(1000+), I found that majority > of the strongly referenced objects on heap are coming from ArrivalWindow. > Currently ArrivalWindow uses BoundedStatsDeque which uses > LinkedBlockingDeque deque. > For a cluster of size 1000, it will translate into 2 million objects. > We can use primitives and use an array of long(long[]). This will cut down on > the number of objects and the change is not that big. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9475) Duplicate compilation of UDFs on coordinator
[ https://issues.apache.org/jira/browse/CASSANDRA-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590706#comment-14590706 ] Tyler Hobbs commented on CASSANDRA-9475: +1 on the second patch > Duplicate compilation of UDFs on coordinator > > > Key: CASSANDRA-9475 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9475 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Robert Stupp >Assignee: Robert Stupp >Priority: Minor > Fix For: 2.2.x > > Attachments: 9475-2.txt, 9475.txt > > > User-defined functions are compiled twice on the node actually executing the > CREATE FUNCTION statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9496) ArrivalWindow should use primitives
[ https://issues.apache.org/jira/browse/CASSANDRA-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590665#comment-14590665 ] Jason Brown commented on CASSANDRA-9496: +1, looks good. I also took the liberty of removing BoundedStatsDeque and it's test as they are no longer used after this change. Commited to 2.0, 2.1, 2.2, and trunk. Thanks! > ArrivalWindow should use primitives > > > Key: CASSANDRA-9496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9496 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Minor > Attachments: 9496.txt > > > While doing a heap analysis of a large cluster(1000+), I found that majority > of the strongly referenced objects on heap are coming from ArrivalWindow. > Currently ArrivalWindow uses BoundedStatsDeque which uses > LinkedBlockingDeque deque. > For a cluster of size 1000, it will translate into 2 million objects. > We can use primitives and use an array of long(long[]). This will cut down on > the number of objects and the change is not that big. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[4/4] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/72ab4b94 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/72ab4b94 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/72ab4b94 Branch: refs/heads/trunk Commit: 72ab4b944f915761d8b1bfd22ff75200499e6718 Parents: 075ff50 8b021db Author: Jason Brown Authored: Wed Jun 17 14:39:43 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:39:43 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 +++-- .../cassandra/utils/BoundedStatsDeque.java | 68 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 --- 5 files changed, 111 insertions(+), 140 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/72ab4b94/CHANGES.txt --
[1/4] cassandra git commit: ArrivalWindow should use primitives
Repository: cassandra Updated Branches: refs/heads/trunk 075ff5000 -> 72ab4b944 ArrivalWindow should use primitives patch by sankalp kohli; reviewed by jasobrown for CASSANDRA-9496 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ad8047ab Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ad8047ab Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ad8047ab Branch: refs/heads/trunk Commit: ad8047abdf5db6652b9586e039debb1e855db09a Parents: ec52e77 Author: Jason Brown Authored: Wed Jun 17 14:33:44 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:33:44 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 ++-- .../cassandra/utils/BoundedStatsDeque.java | 72 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 -- 5 files changed, 111 insertions(+), 144 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 6d031f6..753fb1c 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) * Backport indexed value validation fix from CASSANDRA-9057 (CASSANDRA-9564) http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/src/java/org/apache/cassandra/gms/FailureDetector.java -- diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java index e247e48..8fdd99f 100644 --- a/src/java/org/apache/cassandra/gms/FailureDetector.java +++ b/src/java/org/apache/cassandra/gms/FailureDetector.java @@ -27,14 +27,12 @@ import java.util.concurrent.TimeUnit; import javax.management.MBeanServer; import javax.management.ObjectName; -import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.cassandra.config.DatabaseDescriptor; import org.apache.cassandra.io.FSWriteError; import org.apache.cassandra.io.util.FileUtils; -import org.apache.cassandra.utils.BoundedStatsDeque; import org.apache.cassandra.utils.FBUtilities; /** @@ -289,11 +287,60 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean } } +/* + This class is not thread safe. + */ +class ArrayBackedBoundedStats +{ +private final long[] arrivalIntervals; +private long sum = 0; +private int index = 0; +private boolean isFilled = false; +private volatile double mean = 0; + +public ArrayBackedBoundedStats(final int size) +{ +arrivalIntervals = new long[size]; +} + +public void add(long interval) +{ +if(index == arrivalIntervals.length) +{ +isFilled = true; +index = 0; +} + +if(isFilled) +sum = sum - arrivalIntervals[index]; + +arrivalIntervals[index++] = interval; +sum += interval; +mean = (double)sum / size(); +} + +private int size() +{ +return isFilled ? arrivalIntervals.length : index; +} + +public double mean() +{ +return mean; +} + +public long[] getArrivalIntervals() +{ +return arrivalIntervals; +} + +} + class ArrivalWindow { private static final Logger logger = LoggerFactory.getLogger(ArrivalWindow.class); private long tLast = 0L; -private final BoundedStatsDeque arrivalIntervals; +private final ArrayBackedBoundedStats arrivalIntervals; // this is useless except to provide backwards compatibility in phi_convict_threshold, // because everyone seems pretty accustomed to the default of 8, and users who have @@ -309,7 +356,7 @@ class ArrivalWindow ArrivalWindow(int size) { -arrivalIntervals = new BoundedStatsDeque(size); +arrivalIntervals = new ArrayBackedBoundedStats(size); } private static long getMaxInterval() @@ -355,14 +402,14 @@ class ArrivalWindow // see CASSANDRA-2597 for an explanation of the math at work here. double phi(long tnow) { -assert arrivalIntervals.size() > 0 && tLast > 0; // should not be called before any samples arrive +assert arrivalIntervals.mean() > 0 && tLast > 0; // should not be called before any samples arrive long t = tnow - tLast; return t / mean(); } public S
[3/4] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8b021db7 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8b021db7 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8b021db7 Branch: refs/heads/trunk Commit: 8b021db7c2b4adbb5db8e8bba3ca916ef41fc4f7 Parents: 6068efb 4c15970 Author: Jason Brown Authored: Wed Jun 17 14:39:09 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:39:09 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 +++-- .../cassandra/utils/BoundedStatsDeque.java | 68 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 --- 5 files changed, 111 insertions(+), 140 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b021db7/CHANGES.txt -- diff --cc CHANGES.txt index 3e9940c,8f3f9f0..c32596c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,40 -1,12 +1,41 @@@ -2.1.7 +2.2 + * Fix connection leak in CqlRecordWriter (CASSANDRA-9576) + * Mlockall before opening system sstables & remove boot_without_jna option (CASSANDRA-9573) + * Add functions to convert timeuuid to date or time, deprecate dateOf and unixTimestampOf (CASSANDRA-9229) + * Make sure we cancel non-compacting sstables from LifecycleTransaction (CASSANDRA-9566) + * Fix deprecated repair JMX API (CASSANDRA-9570) +Merged from 2.1: * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) Merged from 2.0 + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) + * Add logback metrics (CASSANDRA-9378) -2.1.6 +2.2.0-rc1 + * Compressed commit log should measure compressed space used (CASSANDRA-9095) + * Fix comparison bug in CassandraRoleManager#collectRoles (CASSANDRA-9551) + * Add tinyint,smallint,time,date support for UDFs (CASSANDRA-9400) + * Deprecates SSTableSimpleWriter and SSTableSimpleUnsortedWriter (CASSANDRA-9546) + * Empty INITCOND treated as null in aggregate (CASSANDRA-9457) + * Remove use of Cell in Thrift MapReduce classes (CASSANDRA-8609) + * Integrate pre-release Java Driver 2.2-rc1, custom build (CASSANDRA-9493) + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) + * ant test-all results incomplete when parsed (CASSANDRA-9463) + * Disallow frozen<> types in function arguments and return types for + clarity (CASSANDRA-9411) + * Static Analysis to warn on unsafe use of Autocloseable instances (CASSANDRA-9431) + * Update commitlog archiving examples now that commitlog segments are + not recycled (CASSANDRA-9350) + * Extend Transactional API to sstable lifecycle management (CASSANDRA-8568) + * (cqlsh) Add support for native protocol 4 (CASSANDRA-9399) + * Ensure that UDF and UDAs are keyspace-isolated (CASSANDRA-9409) + * Revert CASSANDRA-7807 (tracing completion client notifications) (CASSANDRA-9429) + * Add ability to stop compaction by ID (CASSANDRA-7207) + * Let CassandraVersion handle SNAPSHOT version (CASSANDRA-9438) +Merged from 2.1: * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) * Use ProtocolError code instead of ServerError code for native protocol http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b021db7/src/java/org/apache/cassandra/gms/FailureDetector.java --
[2/4] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/utils/BoundedStatsDeque.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4c159701 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4c159701 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4c159701 Branch: refs/heads/trunk Commit: 4c15970119e021dd0fe4b2fe8b4f9c594d21f334 Parents: 7c5fc40 ad8047a Author: Jason Brown Authored: Wed Jun 17 14:36:59 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:36:59 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 +++-- .../cassandra/utils/BoundedStatsDeque.java | 68 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 --- 5 files changed, 111 insertions(+), 140 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c159701/CHANGES.txt -- diff --cc CHANGES.txt index 009d974,753fb1c..8f3f9f0 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,51 -1,8 +1,52 @@@ -2.0.16: +2.1.7 + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) +Merged from 2.0 + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) - * Backport indexed value validation fix from CASSANDRA-9057 (CASSANDRA-9564) + + +2.1.6 + * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) + * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) + * Use ProtocolError code instead of ServerError code for native protocol + error responses to unsupported protocol versions (CASSANDRA-9451) + * Default commitlog_sync_batch_window_in_ms changed to 2ms (CASSANDRA-9504) + * Fix empty partition assertion in unsorted sstable writing tools (CASSANDRA-9071) + * Ensure truncate without snapshot cannot produce corrupt responses (CASSANDRA-9388) + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: * Don't accumulate more range than necessary in RangeTombstone.Tracker (CASSANDRA-9486) * Add broadcast and rpc addresses to system.local (CASSANDRA-9436) * Always mark sstable suspect when corrupted (CASSANDRA-9478) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c159701/src/java/org/apache/cassandra/gms/FailureDetector.java
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8b021db7 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8b021db7 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8b021db7 Branch: refs/heads/cassandra-2.2 Commit: 8b021db7c2b4adbb5db8e8bba3ca916ef41fc4f7 Parents: 6068efb 4c15970 Author: Jason Brown Authored: Wed Jun 17 14:39:09 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:39:09 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 +++-- .../cassandra/utils/BoundedStatsDeque.java | 68 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 --- 5 files changed, 111 insertions(+), 140 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b021db7/CHANGES.txt -- diff --cc CHANGES.txt index 3e9940c,8f3f9f0..c32596c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,40 -1,12 +1,41 @@@ -2.1.7 +2.2 + * Fix connection leak in CqlRecordWriter (CASSANDRA-9576) + * Mlockall before opening system sstables & remove boot_without_jna option (CASSANDRA-9573) + * Add functions to convert timeuuid to date or time, deprecate dateOf and unixTimestampOf (CASSANDRA-9229) + * Make sure we cancel non-compacting sstables from LifecycleTransaction (CASSANDRA-9566) + * Fix deprecated repair JMX API (CASSANDRA-9570) +Merged from 2.1: * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) Merged from 2.0 + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) + * Add logback metrics (CASSANDRA-9378) -2.1.6 +2.2.0-rc1 + * Compressed commit log should measure compressed space used (CASSANDRA-9095) + * Fix comparison bug in CassandraRoleManager#collectRoles (CASSANDRA-9551) + * Add tinyint,smallint,time,date support for UDFs (CASSANDRA-9400) + * Deprecates SSTableSimpleWriter and SSTableSimpleUnsortedWriter (CASSANDRA-9546) + * Empty INITCOND treated as null in aggregate (CASSANDRA-9457) + * Remove use of Cell in Thrift MapReduce classes (CASSANDRA-8609) + * Integrate pre-release Java Driver 2.2-rc1, custom build (CASSANDRA-9493) + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) + * ant test-all results incomplete when parsed (CASSANDRA-9463) + * Disallow frozen<> types in function arguments and return types for + clarity (CASSANDRA-9411) + * Static Analysis to warn on unsafe use of Autocloseable instances (CASSANDRA-9431) + * Update commitlog archiving examples now that commitlog segments are + not recycled (CASSANDRA-9350) + * Extend Transactional API to sstable lifecycle management (CASSANDRA-8568) + * (cqlsh) Add support for native protocol 4 (CASSANDRA-9399) + * Ensure that UDF and UDAs are keyspace-isolated (CASSANDRA-9409) + * Revert CASSANDRA-7807 (tracing completion client notifications) (CASSANDRA-9429) + * Add ability to stop compaction by ID (CASSANDRA-7207) + * Let CassandraVersion handle SNAPSHOT version (CASSANDRA-9438) +Merged from 2.1: * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) * Use ProtocolError code instead of ServerError code for native protocol http://git-wip-us.apache.org/repos/asf/cassandra/blob/8b021db7/src/java/org/apache/cassandra/gms/FailureDetector.java --
[1/3] cassandra git commit: ArrivalWindow should use primitives
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 6068efbaf -> 8b021db7c ArrivalWindow should use primitives patch by sankalp kohli; reviewed by jasobrown for CASSANDRA-9496 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ad8047ab Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ad8047ab Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ad8047ab Branch: refs/heads/cassandra-2.2 Commit: ad8047abdf5db6652b9586e039debb1e855db09a Parents: ec52e77 Author: Jason Brown Authored: Wed Jun 17 14:33:44 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:33:44 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 ++-- .../cassandra/utils/BoundedStatsDeque.java | 72 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 -- 5 files changed, 111 insertions(+), 144 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 6d031f6..753fb1c 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) * Backport indexed value validation fix from CASSANDRA-9057 (CASSANDRA-9564) http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/src/java/org/apache/cassandra/gms/FailureDetector.java -- diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java index e247e48..8fdd99f 100644 --- a/src/java/org/apache/cassandra/gms/FailureDetector.java +++ b/src/java/org/apache/cassandra/gms/FailureDetector.java @@ -27,14 +27,12 @@ import java.util.concurrent.TimeUnit; import javax.management.MBeanServer; import javax.management.ObjectName; -import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.cassandra.config.DatabaseDescriptor; import org.apache.cassandra.io.FSWriteError; import org.apache.cassandra.io.util.FileUtils; -import org.apache.cassandra.utils.BoundedStatsDeque; import org.apache.cassandra.utils.FBUtilities; /** @@ -289,11 +287,60 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean } } +/* + This class is not thread safe. + */ +class ArrayBackedBoundedStats +{ +private final long[] arrivalIntervals; +private long sum = 0; +private int index = 0; +private boolean isFilled = false; +private volatile double mean = 0; + +public ArrayBackedBoundedStats(final int size) +{ +arrivalIntervals = new long[size]; +} + +public void add(long interval) +{ +if(index == arrivalIntervals.length) +{ +isFilled = true; +index = 0; +} + +if(isFilled) +sum = sum - arrivalIntervals[index]; + +arrivalIntervals[index++] = interval; +sum += interval; +mean = (double)sum / size(); +} + +private int size() +{ +return isFilled ? arrivalIntervals.length : index; +} + +public double mean() +{ +return mean; +} + +public long[] getArrivalIntervals() +{ +return arrivalIntervals; +} + +} + class ArrivalWindow { private static final Logger logger = LoggerFactory.getLogger(ArrivalWindow.class); private long tLast = 0L; -private final BoundedStatsDeque arrivalIntervals; +private final ArrayBackedBoundedStats arrivalIntervals; // this is useless except to provide backwards compatibility in phi_convict_threshold, // because everyone seems pretty accustomed to the default of 8, and users who have @@ -309,7 +356,7 @@ class ArrivalWindow ArrivalWindow(int size) { -arrivalIntervals = new BoundedStatsDeque(size); +arrivalIntervals = new ArrayBackedBoundedStats(size); } private static long getMaxInterval() @@ -355,14 +402,14 @@ class ArrivalWindow // see CASSANDRA-2597 for an explanation of the math at work here. double phi(long tnow) { -assert arrivalIntervals.size() > 0 && tLast > 0; // should not be called before any samples arrive +assert arrivalIntervals.mean() > 0 && tLast > 0; // should not be called before any samples arrive long t = tnow - tLast; return t / mean(); }
[2/3] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/utils/BoundedStatsDeque.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4c159701 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4c159701 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4c159701 Branch: refs/heads/cassandra-2.2 Commit: 4c15970119e021dd0fe4b2fe8b4f9c594d21f334 Parents: 7c5fc40 ad8047a Author: Jason Brown Authored: Wed Jun 17 14:36:59 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:36:59 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 +++-- .../cassandra/utils/BoundedStatsDeque.java | 68 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 --- 5 files changed, 111 insertions(+), 140 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c159701/CHANGES.txt -- diff --cc CHANGES.txt index 009d974,753fb1c..8f3f9f0 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,51 -1,8 +1,52 @@@ -2.0.16: +2.1.7 + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) +Merged from 2.0 + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) - * Backport indexed value validation fix from CASSANDRA-9057 (CASSANDRA-9564) + + +2.1.6 + * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) + * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) + * Use ProtocolError code instead of ServerError code for native protocol + error responses to unsupported protocol versions (CASSANDRA-9451) + * Default commitlog_sync_batch_window_in_ms changed to 2ms (CASSANDRA-9504) + * Fix empty partition assertion in unsorted sstable writing tools (CASSANDRA-9071) + * Ensure truncate without snapshot cannot produce corrupt responses (CASSANDRA-9388) + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: * Don't accumulate more range than necessary in RangeTombstone.Tracker (CASSANDRA-9486) * Add broadcast and rpc addresses to system.local (CASSANDRA-9436) * Always mark sstable suspect when corrupted (CASSANDRA-9478) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c159701/src/java/org/apache/cassandra/gms/FailureDetector.java
[1/2] cassandra git commit: ArrivalWindow should use primitives
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 7c5fc40b8 -> 4c1597011 ArrivalWindow should use primitives patch by sankalp kohli; reviewed by jasobrown for CASSANDRA-9496 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ad8047ab Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ad8047ab Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ad8047ab Branch: refs/heads/cassandra-2.1 Commit: ad8047abdf5db6652b9586e039debb1e855db09a Parents: ec52e77 Author: Jason Brown Authored: Wed Jun 17 14:33:44 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:33:44 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 ++-- .../cassandra/utils/BoundedStatsDeque.java | 72 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 -- 5 files changed, 111 insertions(+), 144 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 6d031f6..753fb1c 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) * Backport indexed value validation fix from CASSANDRA-9057 (CASSANDRA-9564) http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/src/java/org/apache/cassandra/gms/FailureDetector.java -- diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java index e247e48..8fdd99f 100644 --- a/src/java/org/apache/cassandra/gms/FailureDetector.java +++ b/src/java/org/apache/cassandra/gms/FailureDetector.java @@ -27,14 +27,12 @@ import java.util.concurrent.TimeUnit; import javax.management.MBeanServer; import javax.management.ObjectName; -import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.cassandra.config.DatabaseDescriptor; import org.apache.cassandra.io.FSWriteError; import org.apache.cassandra.io.util.FileUtils; -import org.apache.cassandra.utils.BoundedStatsDeque; import org.apache.cassandra.utils.FBUtilities; /** @@ -289,11 +287,60 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean } } +/* + This class is not thread safe. + */ +class ArrayBackedBoundedStats +{ +private final long[] arrivalIntervals; +private long sum = 0; +private int index = 0; +private boolean isFilled = false; +private volatile double mean = 0; + +public ArrayBackedBoundedStats(final int size) +{ +arrivalIntervals = new long[size]; +} + +public void add(long interval) +{ +if(index == arrivalIntervals.length) +{ +isFilled = true; +index = 0; +} + +if(isFilled) +sum = sum - arrivalIntervals[index]; + +arrivalIntervals[index++] = interval; +sum += interval; +mean = (double)sum / size(); +} + +private int size() +{ +return isFilled ? arrivalIntervals.length : index; +} + +public double mean() +{ +return mean; +} + +public long[] getArrivalIntervals() +{ +return arrivalIntervals; +} + +} + class ArrivalWindow { private static final Logger logger = LoggerFactory.getLogger(ArrivalWindow.class); private long tLast = 0L; -private final BoundedStatsDeque arrivalIntervals; +private final ArrayBackedBoundedStats arrivalIntervals; // this is useless except to provide backwards compatibility in phi_convict_threshold, // because everyone seems pretty accustomed to the default of 8, and users who have @@ -309,7 +356,7 @@ class ArrivalWindow ArrivalWindow(int size) { -arrivalIntervals = new BoundedStatsDeque(size); +arrivalIntervals = new ArrayBackedBoundedStats(size); } private static long getMaxInterval() @@ -355,14 +402,14 @@ class ArrivalWindow // see CASSANDRA-2597 for an explanation of the math at work here. double phi(long tnow) { -assert arrivalIntervals.size() > 0 && tLast > 0; // should not be called before any samples arrive +assert arrivalIntervals.mean() > 0 && tLast > 0; // should not be called before any samples arrive long t = tnow - tLast; return t / mean(); }
[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/utils/BoundedStatsDeque.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4c159701 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4c159701 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4c159701 Branch: refs/heads/cassandra-2.1 Commit: 4c15970119e021dd0fe4b2fe8b4f9c594d21f334 Parents: 7c5fc40 ad8047a Author: Jason Brown Authored: Wed Jun 17 14:36:59 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:36:59 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 +++-- .../cassandra/utils/BoundedStatsDeque.java | 68 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 --- 5 files changed, 111 insertions(+), 140 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c159701/CHANGES.txt -- diff --cc CHANGES.txt index 009d974,753fb1c..8f3f9f0 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,51 -1,8 +1,52 @@@ -2.0.16: +2.1.7 + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) +Merged from 2.0 + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) - * Backport indexed value validation fix from CASSANDRA-9057 (CASSANDRA-9564) + + +2.1.6 + * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) + * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) + * Use ProtocolError code instead of ServerError code for native protocol + error responses to unsupported protocol versions (CASSANDRA-9451) + * Default commitlog_sync_batch_window_in_ms changed to 2ms (CASSANDRA-9504) + * Fix empty partition assertion in unsorted sstable writing tools (CASSANDRA-9071) + * Ensure truncate without snapshot cannot produce corrupt responses (CASSANDRA-9388) + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: * Don't accumulate more range than necessary in RangeTombstone.Tracker (CASSANDRA-9486) * Add broadcast and rpc addresses to system.local (CASSANDRA-9436) * Always mark sstable suspect when corrupted (CASSANDRA-9478) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4c159701/src/java/org/apache/cassandra/gms/FailureDetector.java
cassandra git commit: ArrivalWindow should use primitives
Repository: cassandra Updated Branches: refs/heads/cassandra-2.0 ec52e77ec -> ad8047abd ArrivalWindow should use primitives patch by sankalp kohli; reviewed by jasobrown for CASSANDRA-9496 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ad8047ab Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ad8047ab Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ad8047ab Branch: refs/heads/cassandra-2.0 Commit: ad8047abdf5db6652b9586e039debb1e855db09a Parents: ec52e77 Author: Jason Brown Authored: Wed Jun 17 14:33:44 2015 -0700 Committer: Jason Brown Committed: Wed Jun 17 14:33:44 2015 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/gms/FailureDetector.java | 59 ++-- .../cassandra/utils/BoundedStatsDeque.java | 72 .../gms/ArrayBackedBoundedStatsTest.java| 57 .../cassandra/utils/BoundedStatsDequeTest.java | 66 -- 5 files changed, 111 insertions(+), 144 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 6d031f6..753fb1c 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * ArrivalWindow should use primitives (CASSANDRA-9496) * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) * Backport indexed value validation fix from CASSANDRA-9057 (CASSANDRA-9564) http://git-wip-us.apache.org/repos/asf/cassandra/blob/ad8047ab/src/java/org/apache/cassandra/gms/FailureDetector.java -- diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java index e247e48..8fdd99f 100644 --- a/src/java/org/apache/cassandra/gms/FailureDetector.java +++ b/src/java/org/apache/cassandra/gms/FailureDetector.java @@ -27,14 +27,12 @@ import java.util.concurrent.TimeUnit; import javax.management.MBeanServer; import javax.management.ObjectName; -import org.apache.commons.lang3.StringUtils; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.cassandra.config.DatabaseDescriptor; import org.apache.cassandra.io.FSWriteError; import org.apache.cassandra.io.util.FileUtils; -import org.apache.cassandra.utils.BoundedStatsDeque; import org.apache.cassandra.utils.FBUtilities; /** @@ -289,11 +287,60 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean } } +/* + This class is not thread safe. + */ +class ArrayBackedBoundedStats +{ +private final long[] arrivalIntervals; +private long sum = 0; +private int index = 0; +private boolean isFilled = false; +private volatile double mean = 0; + +public ArrayBackedBoundedStats(final int size) +{ +arrivalIntervals = new long[size]; +} + +public void add(long interval) +{ +if(index == arrivalIntervals.length) +{ +isFilled = true; +index = 0; +} + +if(isFilled) +sum = sum - arrivalIntervals[index]; + +arrivalIntervals[index++] = interval; +sum += interval; +mean = (double)sum / size(); +} + +private int size() +{ +return isFilled ? arrivalIntervals.length : index; +} + +public double mean() +{ +return mean; +} + +public long[] getArrivalIntervals() +{ +return arrivalIntervals; +} + +} + class ArrivalWindow { private static final Logger logger = LoggerFactory.getLogger(ArrivalWindow.class); private long tLast = 0L; -private final BoundedStatsDeque arrivalIntervals; +private final ArrayBackedBoundedStats arrivalIntervals; // this is useless except to provide backwards compatibility in phi_convict_threshold, // because everyone seems pretty accustomed to the default of 8, and users who have @@ -309,7 +356,7 @@ class ArrivalWindow ArrivalWindow(int size) { -arrivalIntervals = new BoundedStatsDeque(size); +arrivalIntervals = new ArrayBackedBoundedStats(size); } private static long getMaxInterval() @@ -355,14 +402,14 @@ class ArrivalWindow // see CASSANDRA-2597 for an explanation of the math at work here. double phi(long tnow) { -assert arrivalIntervals.size() > 0 && tLast > 0; // should not be called before any samples arrive +assert arrivalIntervals.mean() > 0 && tLast > 0; // should not be called before any samples arrive long t = tnow - tLast; return t / mean(); }
[jira] [Commented] (CASSANDRA-9119) Nodetool rebuild creates an additional rebuild session even if there is one already running
[ https://issues.apache.org/jira/browse/CASSANDRA-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590641#comment-14590641 ] Joshua McKenzie commented on CASSANDRA-9119: +1 > Nodetool rebuild creates an additional rebuild session even if there is one > already running > --- > > Key: CASSANDRA-9119 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9119 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.0.12.200 > Cassandra 2.1.2.98 >Reporter: Jose Martinez Poblete >Assignee: Yuki Morishita > Fix For: 2.1.x, 2.2.0 rc2 > > > If a nodetool rebuild session is started and the shell session is finished > for whatever reason, a second nodetool rebuild will spawn a second rebuild > filestream > {noformat} > DC2-S1-100-29:~ # ps aux | grep nodetool > root 10304 0.0 0.0 4532 560 pts/3 S+ 05:23 0:00 grep nodetool > dds-user 20946 0.0 0.0 21180 1880 ? S 04:39 0:00 /bin/sh > /usr/share/dse/resources/cassandra/bin/nodetool rebuild group10 < there > is only one rebuild running > DC2-S1-100-29:~ # nodetool netstats | grep -v /var/local/cassandra > Mode: NORMAL > Rebuild 818307b0-d9ba-11e4-8d4c-7bce93ffad70 <-- does this represent one > rebuild? > /10.96.100.22 > Receiving 63 files, 221542605741 bytes total > /10.96.100.26 > Receiving 48 files, 47712285610 bytes total > /10.96.100.25 > /10.96.100.23 > Receiving 57 files, 127515362783 bytes total > /10.96.100.27 > /10.96.100.24 > Rebuild 7bf9fcd0-d9bb-11e4-8d4c-7bce93ffad70 <--- does this represent a > second rebuild? > /10.96.100.25 > /10.96.100.26 > Receiving 56 files, 47717905924 bytes total > /10.96.100.24 > /10.96.100.22 > Receiving 61 files, 221558642440 bytes total > /10.96.100.23 > Receiving 62 files, 127528841272 bytes total > /10.96.100.27 > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed > Commands n/a 0 2151322 > Responses n/a 0 3343981 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9615) Speed up Windows launch scripts
Joshua McKenzie created CASSANDRA-9615: -- Summary: Speed up Windows launch scripts Key: CASSANDRA-9615 URL: https://issues.apache.org/jira/browse/CASSANDRA-9615 Project: Cassandra Issue Type: Improvement Reporter: Joshua McKenzie Assignee: Joshua McKenzie Priority: Minor Fix For: 2.2.x Currently the async callback to start C* on Windows from within ccm is taking upwards of 1.5 to 2 seconds per node due to a variety of somewhat expensive process launches we're doing in there (java version check, async port open checking). Contrast this with a crisp 0-1 ms on linux... Some of that stuff can be cleaned up and sped up which should help both speed up our testing environment and iron out an error that pops up on the port check w/IPv6 (note: node still starts, just complains). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9563) Rename class for DATE type in Java driver
[ https://issues.apache.org/jira/browse/CASSANDRA-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590565#comment-14590565 ] Olivier Michallat commented on CASSANDRA-9563: -- It's merged now. > Rename class for DATE type in Java driver > - > > Key: CASSANDRA-9563 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9563 > Project: Cassandra > Issue Type: Improvement >Reporter: Olivier Michallat >Assignee: Robert Stupp >Priority: Minor > Fix For: 2.2.0 rc2 > > > An early preview of the Java driver 2.2 was provided for inclusion in > Cassandra 2.2.0-rc1. It uses a custom Java type to represent CQL type > {{DATE}}. Currently that Java type is called {{DateWithoutTime}}. > We'd like to rename it to {{LocalDate}}. This would be a breaking change for > Cassandra, because that type is visible from UDF implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
[ https://issues.apache.org/jira/browse/CASSANDRA-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590480#comment-14590480 ] Maitrayee edited comment on CASSANDRA-9610 at 6/17/15 8:15 PM: --- in 1.2.19 yaml I have num_tokens: 256 How did you determine that vnodes are not enabled in 1.2.19? was (Author: maitrayee_c): in 1.2.19 yaml I have num_tokens: 256 How did you determine that vodes are not enabled in 1.2.19? > Increased response time with cassandra 2.0.9 from 1.2.19 > > > Key: CASSANDRA-9610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 > Project: Cassandra > Issue Type: Bug >Reporter: Maitrayee > Attachments: servicedefinition_schema.txt, traceout_1.2.19, > traceout_2.0.9 > > > I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with > secondary index was completing much faster in 1.2.19 > Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
[ https://issues.apache.org/jira/browse/CASSANDRA-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590480#comment-14590480 ] Maitrayee edited comment on CASSANDRA-9610 at 6/17/15 8:14 PM: --- in 1.2.19 yaml I have num_tokens: 256 How did you determine that vodes are not enabled in 1.2.19? was (Author: maitrayee_c): in 1.2.19 yaml I have num_tokens: 256 How did you determine that nvodes are not enabled in 1.2.19? > Increased response time with cassandra 2.0.9 from 1.2.19 > > > Key: CASSANDRA-9610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 > Project: Cassandra > Issue Type: Bug >Reporter: Maitrayee > Attachments: servicedefinition_schema.txt, traceout_1.2.19, > traceout_2.0.9 > > > I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with > secondary index was completing much faster in 1.2.19 > Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
[ https://issues.apache.org/jira/browse/CASSANDRA-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590480#comment-14590480 ] Maitrayee commented on CASSANDRA-9610: -- in 1.2.19 yaml I have num_tokens: 256 How did you determine that nvodes are not enabled in 1.2.19? > Increased response time with cassandra 2.0.9 from 1.2.19 > > > Key: CASSANDRA-9610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 > Project: Cassandra > Issue Type: Bug >Reporter: Maitrayee > Attachments: servicedefinition_schema.txt, traceout_1.2.19, > traceout_2.0.9 > > > I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with > secondary index was completing much faster in 1.2.19 > Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9614) I am trying to execute queries using execute_concurrent() in python but I am getting the best throughput on a concurrency of 4 whereas according to the documentation
Kumar Saras created CASSANDRA-9614: -- Summary: I am trying to execute queries using execute_concurrent() in python but I am getting the best throughput on a concurrency of 4 whereas according to the documentation there can be 100 concurrent requests. Why this kind of behavior? Key: CASSANDRA-9614 URL: https://issues.apache.org/jira/browse/CASSANDRA-9614 Project: Cassandra Issue Type: Test Components: Documentation & website Environment: Linux Reporter: Kumar Saras Fix For: 1.2.x -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9601) Allow an initial connection timeout to be set in cqlsh
[ https://issues.apache.org/jira/browse/CASSANDRA-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9601: -- Fix Version/s: (was: 2.2.0 rc2) 2.2.x > Allow an initial connection timeout to be set in cqlsh > -- > > Key: CASSANDRA-9601 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9601 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Mike Adamson >Assignee: Stefania > Labels: cqlsh > Fix For: 2.2.x > > > [PYTHON-206|https://datastax-oss.atlassian.net/browse/PYTHON-206] introduced > the ability to change the initial connection timeout on connections from the > default of 5s. > This change was introduced because some auth providers (kerberos) can take > longer than 5s to complete a first time negotiation for a connection. > cqlsh should allow this setting to be changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9496) ArrivalWindow should use primitives
[ https://issues.apache.org/jira/browse/CASSANDRA-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-9496: --- Reviewer: Jason Brown > ArrivalWindow should use primitives > > > Key: CASSANDRA-9496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9496 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Minor > Attachments: 9496.txt > > > While doing a heap analysis of a large cluster(1000+), I found that majority > of the strongly referenced objects on heap are coming from ArrivalWindow. > Currently ArrivalWindow uses BoundedStatsDeque which uses > LinkedBlockingDeque deque. > For a cluster of size 1000, it will translate into 2 million objects. > We can use primitives and use an array of long(long[]). This will cut down on > the number of objects and the change is not that big. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
[ https://issues.apache.org/jira/browse/CASSANDRA-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590403#comment-14590403 ] Tyler Hobbs commented on CASSANDRA-9610: It looks like your 1.2 cluster does not have vnodes enabled (num_tokens = 1) and the 2.0 cluster does. In 2.1 we've improved the performance when vnodes are enabled with CASSANDRA-1337, but 2.0 doesn't have this improvement. I suggest these options: * Structure your data model so that you don't rely on secondary indexes for queries where performance matters, * Upgrade to 2.1, or * Don't enable vnodes in 2.0 > Increased response time with cassandra 2.0.9 from 1.2.19 > > > Key: CASSANDRA-9610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 > Project: Cassandra > Issue Type: Bug >Reporter: Maitrayee > Attachments: servicedefinition_schema.txt, traceout_1.2.19, > traceout_2.0.9 > > > I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with > secondary index was completing much faster in 1.2.19 > Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
[ https://issues.apache.org/jira/browse/CASSANDRA-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590383#comment-14590383 ] Maitrayee commented on CASSANDRA-9610: -- Attached traceout and schema definition. 1.2.19 and 2.0.9 trace outs are from 2 different env (more nodes in 1.2.19 cluster) > Increased response time with cassandra 2.0.9 from 1.2.19 > > > Key: CASSANDRA-9610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 > Project: Cassandra > Issue Type: Bug >Reporter: Maitrayee > Attachments: servicedefinition_schema.txt, traceout_1.2.19, > traceout_2.0.9 > > > I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with > secondary index was completing much faster in 1.2.19 > Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
[ https://issues.apache.org/jira/browse/CASSANDRA-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maitrayee updated CASSANDRA-9610: - Attachment: traceout_2.0.9 traceout_1.2.19 servicedefinition_schema.txt > Increased response time with cassandra 2.0.9 from 1.2.19 > > > Key: CASSANDRA-9610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 > Project: Cassandra > Issue Type: Bug >Reporter: Maitrayee > Attachments: servicedefinition_schema.txt, traceout_1.2.19, > traceout_2.0.9 > > > I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with > secondary index was completing much faster in 1.2.19 > Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9532) Provide access to select statement's real column definitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590358#comment-14590358 ] Sam Tunnicliffe commented on CASSANDRA-9532: Initially I was also keen on encapsulating the column definitions, however there are a couple of things which complicate that. * Firstly, aliases belong to {{RawSelector}}, so we would need to extract those and pass them along with the list of prepared {{Selectable}} to {{createFactoriesAndCollectColumnDefinitions}} if we are to change that to collate the {{SelectionColumnMapping}}. This itself is slightly complicated by the recursive call in {{WithFunction.newSelectorFactory}}, as here {{args}} are already {{Selectable}} instances. In fact, function arguments cannot be aliased so it's pretty straightforward to fake up a list of null aliases here but it it is a bit ugly and so does reduce the net benefit of the refactoring slightly. * The second issue, also related to the recursive call, is that we would also collect a mapping & column definition on each recursive call, whereas with the current patch we only collate for the top-level factories. To illustrate, a query like {{SELECT ks.function1(col1) FROM ks.table1}} should generate a single mapping, {{\[ks.function1 -> \[col1\]\]}}. If we combine the {{selectorFactory}} creation with the collation though, the recursive call in {{WithFunction.newSelectorFactory}} which processes the args list would generate another mapping from the {{ColumnIdentifier}} argument, so we'd end up with {{\[ks.function1 -> \[col1\], col1 -> \[col1\]\]}}. Again, this could be fixed with a flag to indicate whether to add stuff to the mapping, but I'm not sure if those two workarounds would destroy the value of refactoring in the first place. On the other points, I totally agree and have pushed commits to my dev branches incorporating them (& rebased the dev branches too). > Provide access to select statement's real column definitions > > > Key: CASSANDRA-9532 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9532 > Project: Cassandra > Issue Type: Improvement >Reporter: mck >Assignee: Sam Tunnicliffe > Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x > > Attachments: 9532-2.0-v2.txt, 9532-2.1-v2.txt, 9532-2.2-v2.txt, > 9532-trunk-v2.txt, cassandra-2.0-9532.txt, cassandra-2.1-9532.txt, > cassandra-2.2-9532.txt, trunk-9532.txt > > > Currently there is no way to get access to the real ColumnDefinitions being > used in a SelectStatement. > This information is there in > {{selectStatement.selection.columns}} but is private. > Giving public access would make it possible for third-party implementations > of a {{QueryHandler}} to work accurately with the real columns being queried > and not have to work-around column aliases (or when the rawSelectors don't > map directly to ColumnDefinitions, eg in Selection.fromSelectors(..), like > functions), which is what one has to do today with going through > ResultSet.metadata.names. > This issue provides a very minimal patch to provide access to the already > final and immutable fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9496) ArrivalWindow should use primitives
[ https://issues.apache.org/jira/browse/CASSANDRA-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sankalp kohli updated CASSANDRA-9496: - Attachment: 9496.txt > ArrivalWindow should use primitives > > > Key: CASSANDRA-9496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9496 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Minor > Attachments: 9496.txt > > > While doing a heap analysis of a large cluster(1000+), I found that majority > of the strongly referenced objects on heap are coming from ArrivalWindow. > Currently ArrivalWindow uses BoundedStatsDeque which uses > LinkedBlockingDeque deque. > For a cluster of size 1000, it will translate into 2 million objects. > We can use primitives and use an array of long(long[]). This will cut down on > the number of objects and the change is not that big. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9496) ArrivalWindow should use primitives
[ https://issues.apache.org/jira/browse/CASSANDRA-9496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590351#comment-14590351 ] sankalp kohli commented on CASSANDRA-9496: -- The patch is against 2.0 but I think it should work for later branches. > ArrivalWindow should use primitives > > > Key: CASSANDRA-9496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9496 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: sankalp kohli >Priority: Minor > Attachments: 9496.txt > > > While doing a heap analysis of a large cluster(1000+), I found that majority > of the strongly referenced objects on heap are coming from ArrivalWindow. > Currently ArrivalWindow uses BoundedStatsDeque which uses > LinkedBlockingDeque deque. > For a cluster of size 1000, it will translate into 2 million objects. > We can use primitives and use an array of long(long[]). This will cut down on > the number of objects and the change is not that big. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9613) Omit (de)serialization of state variable in UDAs
Robert Stupp created CASSANDRA-9613: --- Summary: Omit (de)serialization of state variable in UDAs Key: CASSANDRA-9613 URL: https://issues.apache.org/jira/browse/CASSANDRA-9613 Project: Cassandra Issue Type: Improvement Reporter: Robert Stupp Priority: Minor Currently the result of each UDA's state function call is serialized and then deserialized for the next state-function invocation and optionally final function invocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8303) Create a capability limitation framework
[ https://issues.apache.org/jira/browse/CASSANDRA-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590255#comment-14590255 ] Wei Deng edited comment on CASSANDRA-8303 at 6/17/15 6:29 PM: -- I'd like to add one more use case here: we need to allow the DBA to restrict the application code from enabling query tracing. We know tracing is used for debugging query performance problems and can be very useful in the development stage for the Cassandra application developers and DBAs. However, if the developer is not careful to turn it off when they roll out the application code in production, then they might unknowingly incur performance overhead from query tracing. Sometimes this could be overwhelming and you will start to see a lot of dropped _TRACE messages in "nodetool tpstats". The DBA needs to have this control to turn off the capability of enabling query tracing on all tables so that they don't have to worry about something they don't have control (if the query tracing is enabled from the C* driver) but can negatively impact the C* performance. was (Author: weideng): I'd like to add one more use case here: we need to allow the DBA to restrict the application code from enabling query tracing. We know tracing is used for debugging query performance problems and can be very useful in the development stage for the Cassandra application developers and DBAs. However, if the developer is not careful to turn it off when they roll out the application code in production, then they might unknowingly incur performance overhead from query tracing. Sometimes this could be overwhelming and you will start to see a lot of dropped _TRACE messages in "nodetool tpstats". The DBA needs to have this control to turn off the capability of enabling query tracing on all tables so that they don't have to worry about something they don't have control but can negatively impact the C* performance. > Create a capability limitation framework > > > Key: CASSANDRA-8303 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8303 > Project: Cassandra > Issue Type: Improvement >Reporter: Anupam Arora >Assignee: Sam Tunnicliffe > Fix For: 3.x > > > In addition to our current Auth framework that acts as a white list, and > regulates access to data, functions, and roles, it would be beneficial to > have a different, capability limitation framework, that would be orthogonal > to Auth, and would act as a blacklist. > Example uses: > - take away the ability to TRUNCATE from all users but the admin (TRUNCATE > itself would still require MODIFY permission) > - take away the ability to use ALLOW FILTERING from all users but > Spark/Hadoop (SELECT would still require SELECT permission) > - take away the ability to use UNLOGGED BATCH from everyone (the operation > itself would still require MODIFY permission) > - take away the ability to use certain consistency levels (make certain > tables LWT-only for all users, for example) > Original description: > Please provide a "strict mode" option in cassandra that will kick out any CQL > queries that are expensive, e.g. any query with ALLOWS FILTERING, > multi-partition queries, secondary index queries, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9612) Assertion error while running `nodetool cfstats`
mlowicki created CASSANDRA-9612: --- Summary: Assertion error while running `nodetool cfstats` Key: CASSANDRA-9612 URL: https://issues.apache.org/jira/browse/CASSANDRA-9612 Project: Cassandra Issue Type: Bug Environment: C* 2.1.6 Reporter: mlowicki > nodetool cfstats sync.entity {code} Keyspace: sync Read Count: 2916573 Read Latency: 0.26340278573517617 ms. Write Count: 2356495 Write Latency: 0.03296340242606074 ms. Pending Flushes: 0 Table: entity SSTable count: 919 SSTables in each level: [50/4, 11/10, 101/100, 756, 0, 0, 0, 0, 0] Space used (live): 146265014558 Space used (total): 146265014558 Space used by snapshots (total): 0 Off heap memory used (total): 97950899 SSTable Compression Ratio: 0.1870809135227128 error: /var/lib/cassandra/data2/sync/entity-f73d1360770e11e49f1d673dc3e50a5f/sync-entity-tmplink-ka-516810-Data.db -- StackTrace -- java.lang.AssertionError: /var/lib/cassandra/data2/sync/entity-f73d1360770e11e49f1d673dc3e50a5f/sync-entity-tmplink-ka-516810-Data.db at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:270) at org.apache.cassandra.metrics.ColumnFamilyMetrics$9.value(ColumnFamilyMetrics.java:296) at org.apache.cassandra.metrics.ColumnFamilyMetrics$9.value(ColumnFamilyMetrics.java:290) at com.yammer.metrics.reporting.JmxReporter$Gauge.getValue(JmxReporter.java:63) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1464) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420) at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:657) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$2.run(Transport.java:202) at sun.rmi.transport.Transport$2.run(Transport.java:199) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:198) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:567) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(TCPTransport.java:619) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:684) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(TCPTransport.java:681) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:681) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
[ https://issues.apache.org/jira/browse/CASSANDRA-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590262#comment-14590262 ] Tyler Hobbs commented on CASSANDRA-9610: Can you post the schema, the query, the tracing information, and what your response times are for 1.2.19 and 2.0.9? > Increased response time with cassandra 2.0.9 from 1.2.19 > > > Key: CASSANDRA-9610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 > Project: Cassandra > Issue Type: Bug >Reporter: Maitrayee > > I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with > secondary index was completing much faster in 1.2.19 > Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8303) Create a capability limitation framework
[ https://issues.apache.org/jira/browse/CASSANDRA-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590255#comment-14590255 ] Wei Deng commented on CASSANDRA-8303: - I'd like to add one more use case here: we need to allow the DBA to restrict the application code from enabling query tracing. We know tracing is used for debugging query performance problems and can be very useful in the development stage for the Cassandra application developers and DBAs. However, if the developer is not careful to turn it off when they roll out the application code in production, then they might unknowingly incur performance overhead from query tracing. Sometimes this could be overwhelming and you will start to see a lot of dropped _TRACE messages in "nodetool tpstats". The DBA needs to have this control to turn off the capability of enabling query tracing on all tables so that they don't have to worry about something they don't have control but can negatively impact the C* performance. > Create a capability limitation framework > > > Key: CASSANDRA-8303 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8303 > Project: Cassandra > Issue Type: Improvement >Reporter: Anupam Arora >Assignee: Sam Tunnicliffe > Fix For: 3.x > > > In addition to our current Auth framework that acts as a white list, and > regulates access to data, functions, and roles, it would be beneficial to > have a different, capability limitation framework, that would be orthogonal > to Auth, and would act as a blacklist. > Example uses: > - take away the ability to TRUNCATE from all users but the admin (TRUNCATE > itself would still require MODIFY permission) > - take away the ability to use ALLOW FILTERING from all users but > Spark/Hadoop (SELECT would still require SELECT permission) > - take away the ability to use UNLOGGED BATCH from everyone (the operation > itself would still require MODIFY permission) > - take away the ability to use certain consistency levels (make certain > tables LWT-only for all users, for example) > Original description: > Please provide a "strict mode" option in cassandra that will kick out any CQL > queries that are expensive, e.g. any query with ALLOWS FILTERING, > multi-partition queries, secondary index queries, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-9609) org.apache.cassandra.db.marshal.DateType compares dates with negative timestamp incorrectly
[ https://issues.apache.org/jira/browse/CASSANDRA-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Hobbs resolved CASSANDRA-9609. Resolution: Duplicate We fixed this in CASSANDRA-5723 by introducing the TimestampType and switching the CQL {{timestamp}} type to that by default. > org.apache.cassandra.db.marshal.DateType compares dates with negative > timestamp incorrectly > --- > > Key: CASSANDRA-9609 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9609 > Project: Cassandra > Issue Type: Bug >Reporter: Alexander Troshanin > > We got issues in our application. > We have column family with column names as dates. > when we try to search these columns with some criteria like > {code} > // pseudocode > colName < '01.01.2015' > {code} > then we dont receive columns whos names are less then 01.01.1970, i.e. with > negative timestamp. > After a small research we found out, that probably DateType class has a bug > in compateTo method. > Here is very small example shat shows incorrect work of DateType > {code} > import org.apache.cassandra.db.marshal.DateType; > import org.apache.cassandra.db.marshal.LongType; > import org.apache.cassandra.serializers.LongSerializer; > import org.apache.cassandra.serializers.TimestampSerializer; > import java.util.Date; > import java.util.Locale; > import java.util.TimeZone; > public class TestCassandraCompare { > static TimestampSerializer dateSerializer = TimestampSerializer.instance; > static LongSerializer longSerializer = LongSerializer.instance; > static DateType dateType = DateType.instance; > static LongType longType = LongType.instance; > public static void main(String[] args) { > Locale.setDefault(Locale.ENGLISH); > TimeZone.setDefault(TimeZone.getTimeZone("GMT0")); > // check longs > doLongsCheck(10, 20); > doLongsCheck(20, 10); > doLongsCheck(-10, -20); > doLongsCheck(-20, -10); > doLongsCheck(-10, 20); > doLongsCheck(20, -10); > doLongsCheck(10, -20); > doLongsCheck(-20, 10); > // check dates > doDatesCheck(new Date(10), new Date(20), 1); > doDatesCheck(new Date(20), new Date(10), 2); > doDatesCheck(new Date(-10), new Date(-20), 3); > doDatesCheck(new Date(-20), new Date(-10), 4); > doDatesCheck(new Date(-10), new Date(20), 5); > doDatesCheck(new Date(20), new Date(-10), 6); > doDatesCheck(new Date(10), new Date(-20), 7); > doDatesCheck(new Date(-20), new Date(10), 8); > } > private static void doLongsCheck(long l1, long l2) { > int cassandraCompare = longType.compare(longSerializer.serialize(l1), > longSerializer.serialize(l2)); > int javaCoreCompare = Long.compare(l1, l2); > if (cassandraCompare != javaCoreCompare) { > System.err.println("got incorrect result from LongType compare > method." + > "\n\tlong1: " + l1 + > "\n\tlong2: " + l2 + > "\n\tcassandraCompare: " + cassandraCompare + > "\n\tjavaCoreCompare: " + javaCoreCompare > ); > } > } > private static void doDatesCheck(Date d1, Date d2, int testNum) { > int cassandraCompare = dateType.compare(dateSerializer.serialize(d1), > dateSerializer.serialize(d2)); > int javaCoreCompare = d1.compareTo(d2); > if (cassandraCompare != javaCoreCompare) { > System.err.println("[" + testNum + "]got incorrect result from > DateType compare method." + > "\n\tdate1: " + d1 + > "\n\tdate2: " + d2 + > "\n\tcassandraCompare: " + cassandraCompare + > "\n\tjavaCoreCompare: " + javaCoreCompare > ); > } > } > } > {code} > If you will run the code you will se next output: > {code} > [5]got incorrect result from DateType compare method. > date1: Wed Dec 31 23:58:20 GMT 1969 > date2: Thu Jan 01 00:03:20 GMT 1970 > cassandraCompare: 1 > javaCoreCompare: -1 > [6]got incorrect result from DateType compare method. > date1: Thu Jan 01 00:03:20 GMT 1970 > date2: Wed Dec 31 23:58:20 GMT 1969 > cassandraCompare: -1 > javaCoreCompare: 1 > [7]got incorrect result from DateType compare method. > date1: Thu Jan 01 00:01:40 GMT 1970 > date2: Wed Dec 31 23:56:40 GMT 1969 > cassandraCompare: -1 > javaCoreCompare: 1 > [8]got incorrect result from DateType compare method. > date1: Wed Dec 31 23:56:40 GMT 1969 > date2: Thu Jan 01 00:01:40 GMT 1970 > cassandraCompare: 1 > javaC
[jira] [Updated] (CASSANDRA-9609) org.apache.cassandra.db.marshal.DateType compares dates with negative timestamp incorrectly
[ https://issues.apache.org/jira/browse/CASSANDRA-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Troshanin updated CASSANDRA-9609: --- Description: We got issues in our application. We have column family with column names as dates. when we try to search these columns with some criteria like {code} // pseudocode colName < '01.01.2015' {code} then we dont receive columns whos names are less then 01.01.1970, i.e. with negative timestamp. After a small research we found out, that probably DateType class has a bug in compateTo method. Here is very small example shat shows incorrect work of DateType {code} import org.apache.cassandra.db.marshal.DateType; import org.apache.cassandra.db.marshal.LongType; import org.apache.cassandra.serializers.LongSerializer; import org.apache.cassandra.serializers.TimestampSerializer; import java.util.Date; import java.util.Locale; import java.util.TimeZone; public class TestCassandraCompare { static TimestampSerializer dateSerializer = TimestampSerializer.instance; static LongSerializer longSerializer = LongSerializer.instance; static DateType dateType = DateType.instance; static LongType longType = LongType.instance; public static void main(String[] args) { Locale.setDefault(Locale.ENGLISH); TimeZone.setDefault(TimeZone.getTimeZone("GMT0")); // check longs doLongsCheck(10, 20); doLongsCheck(20, 10); doLongsCheck(-10, -20); doLongsCheck(-20, -10); doLongsCheck(-10, 20); doLongsCheck(20, -10); doLongsCheck(10, -20); doLongsCheck(-20, 10); // check dates doDatesCheck(new Date(10), new Date(20), 1); doDatesCheck(new Date(20), new Date(10), 2); doDatesCheck(new Date(-10), new Date(-20), 3); doDatesCheck(new Date(-20), new Date(-10), 4); doDatesCheck(new Date(-10), new Date(20), 5); doDatesCheck(new Date(20), new Date(-10), 6); doDatesCheck(new Date(10), new Date(-20), 7); doDatesCheck(new Date(-20), new Date(10), 8); } private static void doLongsCheck(long l1, long l2) { int cassandraCompare = longType.compare(longSerializer.serialize(l1), longSerializer.serialize(l2)); int javaCoreCompare = Long.compare(l1, l2); if (cassandraCompare != javaCoreCompare) { System.err.println("got incorrect result from LongType compare method." + "\n\tlong1: " + l1 + "\n\tlong2: " + l2 + "\n\tcassandraCompare: " + cassandraCompare + "\n\tjavaCoreCompare: " + javaCoreCompare ); } } private static void doDatesCheck(Date d1, Date d2, int testNum) { int cassandraCompare = dateType.compare(dateSerializer.serialize(d1), dateSerializer.serialize(d2)); int javaCoreCompare = d1.compareTo(d2); if (cassandraCompare != javaCoreCompare) { System.err.println("[" + testNum + "]got incorrect result from DateType compare method." + "\n\tdate1: " + d1 + "\n\tdate2: " + d2 + "\n\tcassandraCompare: " + cassandraCompare + "\n\tjavaCoreCompare: " + javaCoreCompare ); } } } {code} If you will run the code you will se next output: {code} [5]got incorrect result from DateType compare method. date1: Wed Dec 31 23:58:20 GMT 1969 date2: Thu Jan 01 00:03:20 GMT 1970 cassandraCompare: 1 javaCoreCompare: -1 [6]got incorrect result from DateType compare method. date1: Thu Jan 01 00:03:20 GMT 1970 date2: Wed Dec 31 23:58:20 GMT 1969 cassandraCompare: -1 javaCoreCompare: 1 [7]got incorrect result from DateType compare method. date1: Thu Jan 01 00:01:40 GMT 1970 date2: Wed Dec 31 23:56:40 GMT 1969 cassandraCompare: -1 javaCoreCompare: 1 [8]got incorrect result from DateType compare method. date1: Wed Dec 31 23:56:40 GMT 1969 date2: Thu Jan 01 00:01:40 GMT 1970 cassandraCompare: 1 javaCoreCompare: -1 {code} was: We got issues in our application. We have column family with column names as dates. when we try to search these columns with some criteria like {code} // pseudocode colName < '01.01.2015' {code} then we dont receive columns whos names are less then 01.01.1970, i.e. with negative timestamp. After a small research we found out, that probably DateType class has a bug in compateTo method. Here is very small example shat shows incorrect work of DateType {code} import com.netflix.astyanax.serializers.DateSerializer; import com.netflix.astyanax.serializers.LongSerializer; import org.apache.cassandra.db.marshal.DateType; import org.apache.cassandra.db
[jira] [Updated] (CASSANDRA-9605) Error message when creating a KS replicated to a non-existent DC
[ https://issues.apache.org/jira/browse/CASSANDRA-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9605: -- Fix Version/s: 3.x > Error message when creating a KS replicated to a non-existent DC > > > Key: CASSANDRA-9605 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9605 > Project: Cassandra > Issue Type: New Feature >Reporter: Sebastian Estevez >Assignee: Stefania >Priority: Minor > Fix For: 3.x > > > One of the most common mistakes I see with beginners is they mix up the data > center configuration when using network topology strategy because they are > copying from some tutorial or sample code or have recently changed snitches. > This should not be legal: > {code} > create KEYSPACE test1 WITH replication = {'class': > 'NetworkTopologyStrategy', 'doesnotexist': 1} ; > cqlsh> desc KEYSPACE test1 ; > CREATE KEYSPACE test1 WITH replication = {'class': 'NetworkTopologyStrategy', > 'doesnotexist': '1'} AND durable_writes = true; > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9605) Error message when creating a KS replicated to a non-existent DC
[ https://issues.apache.org/jira/browse/CASSANDRA-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9605: -- Issue Type: New Feature (was: Bug) > Error message when creating a KS replicated to a non-existent DC > > > Key: CASSANDRA-9605 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9605 > Project: Cassandra > Issue Type: New Feature >Reporter: Sebastian Estevez >Assignee: Stefania >Priority: Minor > Fix For: 3.x > > > One of the most common mistakes I see with beginners is they mix up the data > center configuration when using network topology strategy because they are > copying from some tutorial or sample code or have recently changed snitches. > This should not be legal: > {code} > create KEYSPACE test1 WITH replication = {'class': > 'NetworkTopologyStrategy', 'doesnotexist': 1} ; > cqlsh> desc KEYSPACE test1 ; > CREATE KEYSPACE test1 WITH replication = {'class': 'NetworkTopologyStrategy', > 'doesnotexist': '1'} AND durable_writes = true; > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9605) Error message when creating a KS replicated to a non-existent DC
[ https://issues.apache.org/jira/browse/CASSANDRA-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9605: -- Assignee: Stefania > Error message when creating a KS replicated to a non-existent DC > > > Key: CASSANDRA-9605 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9605 > Project: Cassandra > Issue Type: Bug >Reporter: Sebastian Estevez >Assignee: Stefania >Priority: Minor > Fix For: 3.x > > > One of the most common mistakes I see with beginners is they mix up the data > center configuration when using network topology strategy because they are > copying from some tutorial or sample code or have recently changed snitches. > This should not be legal: > {code} > create KEYSPACE test1 WITH replication = {'class': > 'NetworkTopologyStrategy', 'doesnotexist': 1} ; > cqlsh> desc KEYSPACE test1 ; > CREATE KEYSPACE test1 WITH replication = {'class': 'NetworkTopologyStrategy', > 'doesnotexist': '1'} AND durable_writes = true; > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9611) wikipedia demo behaves different depending on how the demos .sh's are called
[ https://issues.apache.org/jira/browse/CASSANDRA-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590041#comment-14590041 ] Jon Moses commented on CASSANDRA-9611: -- Whups, my bad. Too many jira tabs open. > wikipedia demo behaves different depending on how the demos .sh's are called > > > Key: CASSANDRA-9611 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9611 > Project: Cassandra > Issue Type: Bug >Reporter: Jon Moses >Priority: Trivial > > If you run the "1-add-schema.sh" script from a directory other that the > wikipedia demo directory, it will "succeed" but be unable to find the > create_table.cql file. Subsequently, setting the solr options will create > the table, but with COMPACT STORAGE, rather than via cql. > Outside dir: > {noformat} > $ ./wikipedia/1-add-schema.sh > Creating Cassandra table... > ./wikipedia/1-add-schema.sh: line 15: create_table.cql: No such file or > directory > Posting solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml... > SUCCESS > Posted solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml > Posting schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml... > SUCCESS > Posted schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml > Creating index... > > > 0 name="QTime">5620 > > Created index. > {noformat} > {noformat} > $ cqlsh -e "desc table wiki.solr" > /* > Warning: Table wiki.solr omitted because it has constructs not compatible > with CQL (was created via legacy API). > Approximate structure, for reference: > (this should not be used to reproduce this schema) > CREATE TABLE wiki.solr ( > key text PRIMARY KEY, > "_docBoost" text, > body text, > date text, > name text, > solr_query text, > title text > ) WITH COMPACT STORAGE > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '4', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = 'NONE'; > CREATE CUSTOM INDEX wiki_solr__docBoost_index ON wiki.solr ("_docBoost") > USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_body_index ON wiki.solr (body) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_date_index ON wiki.solr (date) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_name_index ON wiki.solr (name) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_solr_query_index ON wiki.solr (solr_query) > USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_title_index ON wiki.solr (title) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE TRIGGER dse_augmentation ON wiki.solr USING > 'com.datastax.bdp.search.solr.triggers.SolrAugmentationTrigger'; > */ > {noformat} > Inside dir: > {noformat} > $ ./1-add-schema.sh > Creating Cassandra table... > Posting solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml... > SUCCESS > Posted solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml > Posting schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml... > SUCCESS > Posted schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml > Creating index... > > > 0 name="QTime">1272 > > Created index. > {noformat} > {noformat} > $ cqlsh -e "desc table wiki.solr" > CREATE TABLE wiki.solr ( > id text PRIMARY KEY, > body text, > date text, > name text, > solr_query text, > title text > ) WITH bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '4', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_
[jira] [Resolved] (CASSANDRA-9611) wikipedia demo behaves different depending on how the demos .sh's are called
[ https://issues.apache.org/jira/browse/CASSANDRA-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Stupp resolved CASSANDRA-9611. - Resolution: Invalid The demo is specific to DSE (Search) and not related to OSS Apache Cassandra. [~jmoses] please forward this to DSE Search team. > wikipedia demo behaves different depending on how the demos .sh's are called > > > Key: CASSANDRA-9611 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9611 > Project: Cassandra > Issue Type: Bug >Reporter: Jon Moses >Priority: Trivial > > If you run the "1-add-schema.sh" script from a directory other that the > wikipedia demo directory, it will "succeed" but be unable to find the > create_table.cql file. Subsequently, setting the solr options will create > the table, but with COMPACT STORAGE, rather than via cql. > Outside dir: > {noformat} > $ ./wikipedia/1-add-schema.sh > Creating Cassandra table... > ./wikipedia/1-add-schema.sh: line 15: create_table.cql: No such file or > directory > Posting solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml... > SUCCESS > Posted solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml > Posting schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml... > SUCCESS > Posted schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml > Creating index... > > > 0 name="QTime">5620 > > Created index. > {noformat} > {noformat} > $ cqlsh -e "desc table wiki.solr" > /* > Warning: Table wiki.solr omitted because it has constructs not compatible > with CQL (was created via legacy API). > Approximate structure, for reference: > (this should not be used to reproduce this schema) > CREATE TABLE wiki.solr ( > key text PRIMARY KEY, > "_docBoost" text, > body text, > date text, > name text, > solr_query text, > title text > ) WITH COMPACT STORAGE > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '4', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = 'NONE'; > CREATE CUSTOM INDEX wiki_solr__docBoost_index ON wiki.solr ("_docBoost") > USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_body_index ON wiki.solr (body) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_date_index ON wiki.solr (date) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_name_index ON wiki.solr (name) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_solr_query_index ON wiki.solr (solr_query) > USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE CUSTOM INDEX wiki_solr_title_index ON wiki.solr (title) USING > 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; > CREATE TRIGGER dse_augmentation ON wiki.solr USING > 'com.datastax.bdp.search.solr.triggers.SolrAugmentationTrigger'; > */ > {noformat} > Inside dir: > {noformat} > $ ./1-add-schema.sh > Creating Cassandra table... > Posting solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml... > SUCCESS > Posted solrconfig.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml > Posting schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml... > SUCCESS > Posted schema.xml to > http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml > Creating index... > > > 0 name="QTime">1272 > > Created index. > {noformat} > {noformat} > $ cqlsh -e "desc table wiki.solr" > CREATE TABLE wiki.solr ( > id text PRIMARY KEY, > body text, > date text, > name text, > solr_query text, > title text > ) WITH bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '4', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} >
[jira] [Created] (CASSANDRA-9611) wikipedia demo behaves different depending on how the demos .sh's are called
Jon Moses created CASSANDRA-9611: Summary: wikipedia demo behaves different depending on how the demos .sh's are called Key: CASSANDRA-9611 URL: https://issues.apache.org/jira/browse/CASSANDRA-9611 Project: Cassandra Issue Type: Bug Reporter: Jon Moses Priority: Trivial If you run the "1-add-schema.sh" script from a directory other that the wikipedia demo directory, it will "succeed" but be unable to find the create_table.cql file. Subsequently, setting the solr options will create the table, but with COMPACT STORAGE, rather than via cql. Outside dir: {noformat} $ ./wikipedia/1-add-schema.sh Creating Cassandra table... ./wikipedia/1-add-schema.sh: line 15: create_table.cql: No such file or directory Posting solrconfig.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml... SUCCESS Posted solrconfig.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml Posting schema.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml... SUCCESS Posted schema.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml Creating index... 05620 Created index. {noformat} {noformat} $ cqlsh -e "desc table wiki.solr" /* Warning: Table wiki.solr omitted because it has constructs not compatible with CQL (was created via legacy API). Approximate structure, for reference: (this should not be used to reproduce this schema) CREATE TABLE wiki.solr ( key text PRIMARY KEY, "_docBoost" text, body text, date text, name text, solr_query text, title text ) WITH COMPACT STORAGE AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = 'NONE'; CREATE CUSTOM INDEX wiki_solr__docBoost_index ON wiki.solr ("_docBoost") USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; CREATE CUSTOM INDEX wiki_solr_body_index ON wiki.solr (body) USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; CREATE CUSTOM INDEX wiki_solr_date_index ON wiki.solr (date) USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; CREATE CUSTOM INDEX wiki_solr_name_index ON wiki.solr (name) USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; CREATE CUSTOM INDEX wiki_solr_solr_query_index ON wiki.solr (solr_query) USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; CREATE CUSTOM INDEX wiki_solr_title_index ON wiki.solr (title) USING 'com.datastax.bdp.search.solr.ThriftSolrSecondaryIndex'; CREATE TRIGGER dse_augmentation ON wiki.solr USING 'com.datastax.bdp.search.solr.triggers.SolrAugmentationTrigger'; */ {noformat} Inside dir: {noformat} $ ./1-add-schema.sh Creating Cassandra table... Posting solrconfig.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml... SUCCESS Posted solrconfig.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/solrconfig.xml Posting schema.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml... SUCCESS Posted schema.xml to http://ip-172-31-22-166.ec2.internal:8983/solr/resource/wiki.solr/schema.xml Creating index... 01272 Created index. {noformat} {noformat} $ cqlsh -e "desc table wiki.solr" CREATE TABLE wiki.solr ( id text PRIMARY KEY, body text, date text, name text, solr_query text, title text ) WITH bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; CREATE CUSTOM INDEX wiki_solr_body_index ON wiki.solr (body) USING 'com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex'; CREATE CUSTOM INDEX wiki_solr_date_index ON wiki.solr (date) USING 'com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex'; CREATE CUSTOM INDEX wiki_solr_name_index ON wiki.solr (name) USI
[jira] [Commented] (CASSANDRA-9549) Memory leak
[ https://issues.apache.org/jira/browse/CASSANDRA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1458#comment-1458 ] Benedict commented on CASSANDRA-9549: - Great, glad to hear it > Memory leak > > > Key: CASSANDRA-9549 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9549 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.5. 9 node cluster in EC2 (m1.large nodes, > 2 cores 7.5G memory, 800G platter for cassandra data, root partition and > commit log are on SSD EBS with sufficient IOPS), 3 nodes/availablity zone, 1 > replica/zone > JVM: /usr/java/jdk1.8.0_40/jre/bin/java > JVM Flags besides CP: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar > -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms2G -Xmx2G -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler > -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled > -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark > -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.rmi.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra > -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid > Kernel: Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux >Reporter: Ivar Thorson >Assignee: Benedict >Priority: Critical > Fix For: 2.1.7 > > Attachments: c4_system.log, c7fromboot.zip, cassandra.yaml, > cpu-load.png, memoryuse.png, ref-java-errors.jpeg, suspect.png, two-loads.png > > > We have been experiencing a severe memory leak with Cassandra 2.1.5 that, > over the period of a couple of days, eventually consumes all of the available > JVM heap space, putting the JVM into GC hell where it keeps trying CMS > collection but can't free up any heap space. This pattern happens for every > node in our cluster and is requiring rolling cassandra restarts just to keep > the cluster running. We have upgraded the cluster per Datastax docs from the > 2.0 branch a couple of months ago and have been using the data from this > cluster for more than a year without problem. > As the heap fills up with non-GC-able objects, the CPU/OS load average grows > along with it. Heap dumps reveal an increasing number of > java.util.concurrent.ConcurrentLinkedQueue$Node objects. We took heap dumps > over a 2 day period, and watched the number of Node objects go from 4M, to > 19M, to 36M, and eventually about 65M objects before the node stops > responding. The screen capture of our heap dump is from the 19M measurement. > Load on the cluster is minimal. We can see this effect even with only a > handful of writes per second. (See attachments for Opscenter snapshots during > very light loads and heavier loads). Even with only 5 reads a sec we see this > behavior. > Log files show repeated errors in Ref.java:181 and Ref.java:279 and "LEAK > detected" messages: > {code} > ERROR [CompactionExecutor:557] 2015-06-01 18:27:36,978 Ref.java:279 - Error > when closing class > org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1302301946:/data1/data/ourtablegoeshere-ka-1150 > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32680b31 > rejected from > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@573464d6[Terminated, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1644] > {code} > {code} > ERROR [Reference-Reaper:1] 2015-06-01 18:27:37,083 Ref.java:181 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@74b5df92) to class > org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@2054303604:/data2/data/ourtablegoeshere-ka-1151 > was not released before the reference was garbage collected > {code} > This might be related to [CASSANDRA-8723]? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9609) org.apache.cassandra.db.marshal.DateType compares dates with negative timestamp incorrectly
[ https://issues.apache.org/jira/browse/CASSANDRA-9609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Troshanin updated CASSANDRA-9609: --- Description: We got issues in our application. We have column family with column names as dates. when we try to search these columns with some criteria like {code} // pseudocode colName < '01.01.2015' {code} then we dont receive columns whos names are less then 01.01.1970, i.e. with negative timestamp. After a small research we found out, that probably DateType class has a bug in compateTo method. Here is very small example shat shows incorrect work of DateType {code} import com.netflix.astyanax.serializers.DateSerializer; import com.netflix.astyanax.serializers.LongSerializer; import org.apache.cassandra.db.marshal.DateType; import org.apache.cassandra.db.marshal.LongType; import java.util.Date; import java.util.Locale; import java.util.TimeZone; public class TestCassandraCompare { static DateSerializer dateSerializer = DateSerializer.get(); static LongSerializer longSerializer = LongSerializer.get(); static DateType dateType = DateType.instance; static LongType longType = LongType.instance; public static void main(String[] args) { Locale.setDefault(Locale.ENGLISH); TimeZone.setDefault(TimeZone.getTimeZone("GMT0")); // check longs doLongsCheck(10, 20); doLongsCheck(20, 10); doLongsCheck(-10, -20); doLongsCheck(-20, -10); doLongsCheck(-10, 20); doLongsCheck(20, -10); doLongsCheck(10, -20); doLongsCheck(-20, 10); // check dates doDatesCheck(new Date(10), new Date(20), 1); doDatesCheck(new Date(20), new Date(10), 2); doDatesCheck(new Date(-10), new Date(-20), 3); doDatesCheck(new Date(-20), new Date(-10), 4); doDatesCheck(new Date(-10), new Date(20), 5); doDatesCheck(new Date(20), new Date(-10), 6); doDatesCheck(new Date(10), new Date(-20), 7); doDatesCheck(new Date(-20), new Date(10), 8); } private static void doLongsCheck(long l1, long l2) { int cassandraCompare = longType.compare(longSerializer.toByteBuffer(l1), longSerializer.toByteBuffer(l2)); int javaCoreCompare = Long.compare(l1, l2); if (cassandraCompare != javaCoreCompare) { System.err.println("got incorrect result from LongType compare method." + "\n\tlong1: " + l1 + "\n\tlong2: " + l2 + "\n\tcassandraCompare: " + cassandraCompare + "\n\tjavaCoreCompare: " + javaCoreCompare ); } } private static void doDatesCheck(Date d1, Date d2, int testNum) { int cassandraCompare = dateType.compare(dateSerializer.toByteBuffer(d1), dateSerializer.toByteBuffer(d2)); int javaCoreCompare = d1.compareTo(d2); if (cassandraCompare != javaCoreCompare) { System.err.println("[" + testNum + "]got incorrect result from DateType compare method." + "\n\tdate1: " + d1 + "\n\tdate2: " + d2 + "\n\tcassandraCompare: " + cassandraCompare + "\n\tjavaCoreCompare: " + javaCoreCompare ); } } } {code} If you will run the code you will se next output: {code} [5]got incorrect result from DateType compare method. date1: Wed Dec 31 23:58:20 GMT 1969 date2: Thu Jan 01 00:03:20 GMT 1970 cassandraCompare: 1 javaCoreCompare: -1 [6]got incorrect result from DateType compare method. date1: Thu Jan 01 00:03:20 GMT 1970 date2: Wed Dec 31 23:58:20 GMT 1969 cassandraCompare: -1 javaCoreCompare: 1 [7]got incorrect result from DateType compare method. date1: Thu Jan 01 00:01:40 GMT 1970 date2: Wed Dec 31 23:56:40 GMT 1969 cassandraCompare: -1 javaCoreCompare: 1 [8]got incorrect result from DateType compare method. date1: Wed Dec 31 23:56:40 GMT 1969 date2: Thu Jan 01 00:01:40 GMT 1970 cassandraCompare: 1 javaCoreCompare: -1 {code} was: We got issues in our application. We have column family with column names as dates. when we try to search these columns with some criteria like {code} // pseudocode colName < '01.01.2015' {code} then we dont receive columns whos names are less then 01.01.1970, i.e. with negative timestamp. After a small research we found out, that probably DateType class has a bug in compateTo method. Here is very small example shat shows incorrect work of DateType {code} import com.netflix.astyanax.serializers.DateSerializer; import com.netflix.astyanax.serializers.LongSerializer; import org.apache.cassandra.db.marshal.DateType; import org.apache.cassandra.db.marsha
[jira] [Created] (CASSANDRA-9610) Increased response time with cassandra 2.0.9 from 1.2.19
Maitrayee created CASSANDRA-9610: Summary: Increased response time with cassandra 2.0.9 from 1.2.19 Key: CASSANDRA-9610 URL: https://issues.apache.org/jira/browse/CASSANDRA-9610 Project: Cassandra Issue Type: Bug Reporter: Maitrayee I was using Cassandra 1.2.19. Recently upgraded to 2.0.9. Queries with secondary index was completing much faster in 1.2.19 Validated this with trace on via cqlsh -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9549) Memory leak
[ https://issues.apache.org/jira/browse/CASSANDRA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589997#comment-14589997 ] Ivar Thorson commented on CASSANDRA-9549: - We patched our 2.1.6 cluster on Wednesday and let it run for a day to let things accumulate. Looking at CPU activity and heap space for the last day suggests that the memory leak seems to have been fixed by the patch. Awesome work! > Memory leak > > > Key: CASSANDRA-9549 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9549 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.5. 9 node cluster in EC2 (m1.large nodes, > 2 cores 7.5G memory, 800G platter for cassandra data, root partition and > commit log are on SSD EBS with sufficient IOPS), 3 nodes/availablity zone, 1 > replica/zone > JVM: /usr/java/jdk1.8.0_40/jre/bin/java > JVM Flags besides CP: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar > -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms2G -Xmx2G -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler > -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled > -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark > -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.rmi.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra > -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid > Kernel: Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux >Reporter: Ivar Thorson >Assignee: Benedict >Priority: Critical > Fix For: 2.1.7 > > Attachments: c4_system.log, c7fromboot.zip, cassandra.yaml, > cpu-load.png, memoryuse.png, ref-java-errors.jpeg, suspect.png, two-loads.png > > > We have been experiencing a severe memory leak with Cassandra 2.1.5 that, > over the period of a couple of days, eventually consumes all of the available > JVM heap space, putting the JVM into GC hell where it keeps trying CMS > collection but can't free up any heap space. This pattern happens for every > node in our cluster and is requiring rolling cassandra restarts just to keep > the cluster running. We have upgraded the cluster per Datastax docs from the > 2.0 branch a couple of months ago and have been using the data from this > cluster for more than a year without problem. > As the heap fills up with non-GC-able objects, the CPU/OS load average grows > along with it. Heap dumps reveal an increasing number of > java.util.concurrent.ConcurrentLinkedQueue$Node objects. We took heap dumps > over a 2 day period, and watched the number of Node objects go from 4M, to > 19M, to 36M, and eventually about 65M objects before the node stops > responding. The screen capture of our heap dump is from the 19M measurement. > Load on the cluster is minimal. We can see this effect even with only a > handful of writes per second. (See attachments for Opscenter snapshots during > very light loads and heavier loads). Even with only 5 reads a sec we see this > behavior. > Log files show repeated errors in Ref.java:181 and Ref.java:279 and "LEAK > detected" messages: > {code} > ERROR [CompactionExecutor:557] 2015-06-01 18:27:36,978 Ref.java:279 - Error > when closing class > org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1302301946:/data1/data/ourtablegoeshere-ka-1150 > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32680b31 > rejected from > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@573464d6[Terminated, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1644] > {code} > {code} > ERROR [Reference-Reaper:1] 2015-06-01 18:27:37,083 Ref.java:181 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@74b5df92) to class > org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@2054303604:/data2/data/ourtablegoeshere-ka-1151 > was not released before the reference was garbage collected > {code} > This might be related to [CASSANDRA-8723]? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/3] cassandra git commit: Fix memory leak in Ref due to ConcurrentLinkedQueue behaviour
Repository: cassandra Updated Branches: refs/heads/trunk 5ea5949f9 -> 075ff5000 Fix memory leak in Ref due to ConcurrentLinkedQueue behaviour patch by benedict; reviewed by marcus for CASSANDRA-9549 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7c5fc40b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7c5fc40b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7c5fc40b Branch: refs/heads/trunk Commit: 7c5fc40b8b644e05c32479f2581309f75f981421 Parents: 46cc577 Author: Benedict Elliott Smith Authored: Wed Jun 17 17:02:03 2015 +0100 Committer: Benedict Elliott Smith Committed: Wed Jun 17 17:02:03 2015 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/concurrent/Ref.java | 8 +++- .../cassandra/utils/concurrent/RefCountedTest.java | 15 +++ 3 files changed, 19 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 5f9187c..009d974 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.1.7 + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) Merged from 2.0 * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/src/java/org/apache/cassandra/utils/concurrent/Ref.java -- diff --git a/src/java/org/apache/cassandra/utils/concurrent/Ref.java b/src/java/org/apache/cassandra/utils/concurrent/Ref.java index 4e6cef7..f9876de 100644 --- a/src/java/org/apache/cassandra/utils/concurrent/Ref.java +++ b/src/java/org/apache/cassandra/utils/concurrent/Ref.java @@ -2,12 +2,10 @@ package org.apache.cassandra.utils.concurrent; import java.lang.ref.PhantomReference; import java.lang.ref.ReferenceQueue; +import java.util.Collection; import java.util.Collections; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ConcurrentLinkedQueue; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; +import java.util.concurrent.*; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; @@ -233,7 +231,7 @@ public final class Ref implements RefCounted, AutoCloseable { // we need to retain a reference to each of the PhantomReference instances // we are using to track individual refs -private final ConcurrentLinkedQueue locallyExtant = new ConcurrentLinkedQueue<>(); +private final Collection locallyExtant = new ConcurrentLinkedDeque<>(); // the number of live refs private final AtomicInteger counts = new AtomicInteger(); // the object to call to cleanup when our refs are all finished with http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java -- diff --git a/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java index a9247cd..bb173fe 100644 --- a/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java +++ b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java @@ -21,6 +21,7 @@ package org.apache.cassandra.utils.concurrent; import org.junit.Test; import junit.framework.Assert; +import org.apache.cassandra.utils.ObjectSizes; public class RefCountedTest { @@ -82,4 +83,18 @@ public class RefCountedTest { } } + +@Test +public void testMemoryLeak() +{ +Tidier tidier = new Tidier(); +Ref ref = new Ref(null, tidier); +long initialSize = ObjectSizes.measureDeep(ref); +for (int i = 0 ; i < 1000 ; i++) +ref.ref().release(); +long finalSize = ObjectSizes.measureDeep(ref); +if (finalSize > initialSize * 2) +throw new AssertionError(); +ref.release(); +} }
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Conflicts: CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6068efba Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6068efba Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6068efba Branch: refs/heads/cassandra-2.2 Commit: 6068efbaf43a18c6f9250638bece98010ce59100 Parents: 514dcd9 7c5fc40 Author: Benedict Elliott Smith Authored: Wed Jun 17 17:08:07 2015 +0100 Committer: Benedict Elliott Smith Committed: Wed Jun 17 17:08:07 2015 +0100 -- CHANGES.txt | 4 +++- .../org/apache/cassandra/utils/concurrent/Ref.java | 8 +++- .../cassandra/utils/concurrent/RefCountedTest.java | 15 +++ 3 files changed, 21 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6068efba/CHANGES.txt -- diff --cc CHANGES.txt index 9dccd84,009d974..3e9940c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,38 -1,11 +1,40 @@@ -2.1.7 +2.2 + * Fix connection leak in CqlRecordWriter (CASSANDRA-9576) + * Mlockall before opening system sstables & remove boot_without_jna option (CASSANDRA-9573) + * Add functions to convert timeuuid to date or time, deprecate dateOf and unixTimestampOf (CASSANDRA-9229) + * Make sure we cancel non-compacting sstables from LifecycleTransaction (CASSANDRA-9566) + * Fix deprecated repair JMX API (CASSANDRA-9570) - Merged from 2.0: ++Merged from 2.1: + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) + Merged from 2.0 * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) + * Add logback metrics (CASSANDRA-9378) -2.1.6 +2.2.0-rc1 + * Compressed commit log should measure compressed space used (CASSANDRA-9095) + * Fix comparison bug in CassandraRoleManager#collectRoles (CASSANDRA-9551) + * Add tinyint,smallint,time,date support for UDFs (CASSANDRA-9400) + * Deprecates SSTableSimpleWriter and SSTableSimpleUnsortedWriter (CASSANDRA-9546) + * Empty INITCOND treated as null in aggregate (CASSANDRA-9457) + * Remove use of Cell in Thrift MapReduce classes (CASSANDRA-8609) + * Integrate pre-release Java Driver 2.2-rc1, custom build (CASSANDRA-9493) + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) + * ant test-all results incomplete when parsed (CASSANDRA-9463) + * Disallow frozen<> types in function arguments and return types for + clarity (CASSANDRA-9411) + * Static Analysis to warn on unsafe use of Autocloseable instances (CASSANDRA-9431) + * Update commitlog archiving examples now that commitlog segments are + not recycled (CASSANDRA-9350) + * Extend Transactional API to sstable lifecycle management (CASSANDRA-8568) + * (cqlsh) Add support for native protocol 4 (CASSANDRA-9399) + * Ensure that UDF and UDAs are keyspace-isolated (CASSANDRA-9409) + * Revert CASSANDRA-7807 (tracing completion client notifications) (CASSANDRA-9429) + * Add ability to stop compaction by ID (CASSANDRA-7207) + * Let CassandraVersion handle SNAPSHOT version (CASSANDRA-9438) +Merged from 2.1: * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) * Use ProtocolError code instead of ServerError code for native protocol http://git-wip-us.apache.org/repos/asf/cassandra/blob/6068efba/src/java/org/apache/cassandra/utils/concurrent/Ref.java --
[2/3] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Conflicts: CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6068efba Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6068efba Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6068efba Branch: refs/heads/trunk Commit: 6068efbaf43a18c6f9250638bece98010ce59100 Parents: 514dcd9 7c5fc40 Author: Benedict Elliott Smith Authored: Wed Jun 17 17:08:07 2015 +0100 Committer: Benedict Elliott Smith Committed: Wed Jun 17 17:08:07 2015 +0100 -- CHANGES.txt | 4 +++- .../org/apache/cassandra/utils/concurrent/Ref.java | 8 +++- .../cassandra/utils/concurrent/RefCountedTest.java | 15 +++ 3 files changed, 21 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6068efba/CHANGES.txt -- diff --cc CHANGES.txt index 9dccd84,009d974..3e9940c --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,38 -1,11 +1,40 @@@ -2.1.7 +2.2 + * Fix connection leak in CqlRecordWriter (CASSANDRA-9576) + * Mlockall before opening system sstables & remove boot_without_jna option (CASSANDRA-9573) + * Add functions to convert timeuuid to date or time, deprecate dateOf and unixTimestampOf (CASSANDRA-9229) + * Make sure we cancel non-compacting sstables from LifecycleTransaction (CASSANDRA-9566) + * Fix deprecated repair JMX API (CASSANDRA-9570) - Merged from 2.0: ++Merged from 2.1: + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) + Merged from 2.0 * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) + * Add logback metrics (CASSANDRA-9378) -2.1.6 +2.2.0-rc1 + * Compressed commit log should measure compressed space used (CASSANDRA-9095) + * Fix comparison bug in CassandraRoleManager#collectRoles (CASSANDRA-9551) + * Add tinyint,smallint,time,date support for UDFs (CASSANDRA-9400) + * Deprecates SSTableSimpleWriter and SSTableSimpleUnsortedWriter (CASSANDRA-9546) + * Empty INITCOND treated as null in aggregate (CASSANDRA-9457) + * Remove use of Cell in Thrift MapReduce classes (CASSANDRA-8609) + * Integrate pre-release Java Driver 2.2-rc1, custom build (CASSANDRA-9493) + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) + * ant test-all results incomplete when parsed (CASSANDRA-9463) + * Disallow frozen<> types in function arguments and return types for + clarity (CASSANDRA-9411) + * Static Analysis to warn on unsafe use of Autocloseable instances (CASSANDRA-9431) + * Update commitlog archiving examples now that commitlog segments are + not recycled (CASSANDRA-9350) + * Extend Transactional API to sstable lifecycle management (CASSANDRA-8568) + * (cqlsh) Add support for native protocol 4 (CASSANDRA-9399) + * Ensure that UDF and UDAs are keyspace-isolated (CASSANDRA-9409) + * Revert CASSANDRA-7807 (tracing completion client notifications) (CASSANDRA-9429) + * Add ability to stop compaction by ID (CASSANDRA-7207) + * Let CassandraVersion handle SNAPSHOT version (CASSANDRA-9438) +Merged from 2.1: * (cqlsh) Fix using COPY through SOURCE or -f (CASSANDRA-9083) * Fix occasional lack of `system` keyspace in schema tables (CASSANDRA-8487) * Use ProtocolError code instead of ServerError code for native protocol http://git-wip-us.apache.org/repos/asf/cassandra/blob/6068efba/src/java/org/apache/cassandra/utils/concurrent/Ref.java --
[3/3] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/075ff500 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/075ff500 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/075ff500 Branch: refs/heads/trunk Commit: 075ff5000ced24b42f3b540815cae471bee4049d Parents: 5ea5949 6068efb Author: Benedict Elliott Smith Authored: Wed Jun 17 17:14:28 2015 +0100 Committer: Benedict Elliott Smith Committed: Wed Jun 17 17:14:28 2015 +0100 -- CHANGES.txt | 4 +++- .../org/apache/cassandra/utils/concurrent/Ref.java | 8 +++- .../cassandra/utils/concurrent/RefCountedTest.java | 15 +++ 3 files changed, 21 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/075ff500/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/075ff500/src/java/org/apache/cassandra/utils/concurrent/Ref.java --
[jira] [Created] (CASSANDRA-9609) org.apache.cassandra.db.marshal.DateType compares dates with negative timestamp incorrectly
Alexander Troshanin created CASSANDRA-9609: -- Summary: org.apache.cassandra.db.marshal.DateType compares dates with negative timestamp incorrectly Key: CASSANDRA-9609 URL: https://issues.apache.org/jira/browse/CASSANDRA-9609 Project: Cassandra Issue Type: Bug Reporter: Alexander Troshanin We got issues in our application. We have column family with column names as dates. when we try to search these columns with some criteria like {code} // pseudocode colName < '01.01.2015' {code} then we dont receive columns whos names are less then 01.01.1970, i.e. with negative timestamp. After a small research we found out, that probably DateType class has a bug in compateTo method. Here is very small example shat shows incorrect work of DateType {code} import com.netflix.astyanax.serializers.DateSerializer; import com.netflix.astyanax.serializers.LongSerializer; import org.apache.cassandra.db.marshal.DateType; import org.apache.cassandra.db.marshal.LongType; import java.util.Date; import java.util.Locale; import java.util.TimeZone; public class TestCassandraCompare { static DateSerializer dateSerializer = DateSerializer.get(); static LongSerializer longSerializer = LongSerializer.get(); static DateType dateType = DateType.instance; static LongType longType = LongType.instance; public static void main(String[] args) { Locale.setDefault(Locale.ENGLISH); TimeZone.setDefault(TimeZone.getTimeZone("GMT0")); // check longs doLongsCheck(10, 20); doLongsCheck(20, 10); doLongsCheck(-10, -20); doLongsCheck(-20, -10); doLongsCheck(-10, 20); doLongsCheck(20, -10); doLongsCheck(10, -20); doLongsCheck(-20, 10); // check dates doDatesCheck(new Date(10), new Date(20), 1); doDatesCheck(new Date(20), new Date(10), 2); doDatesCheck(new Date(-10), new Date(-20), 3); doDatesCheck(new Date(-20), new Date(-10), 4); doDatesCheck(new Date(-10), new Date(20), 5); doDatesCheck(new Date(20), new Date(-10), 6); doDatesCheck(new Date(10), new Date(-20), 7); doDatesCheck(new Date(-20), new Date(10), 8); } private static void doLongsCheck(long l1, long l2) { int cassandraCompare = longType.compare(longSerializer.toByteBuffer(l1), longSerializer.toByteBuffer(l2)); int javaCoreCompare = Long.compare(l1, l2); if (cassandraCompare != javaCoreCompare) { System.err.println("got incorrect result from LongType compare method." + "\n\tlong1: " + l1 + "\n\tlong2: " + l2 + "\n\tcassandraCompare: " + cassandraCompare + "\n\tjavaCoreCompare: " + javaCoreCompare ); } } private static void doDatesCheck(Date d1, Date d2, int testNum) { int cassandraCompare = dateType.compare(dateSerializer.toByteBuffer(d1), dateSerializer.toByteBuffer(d2)); int javaCoreCompare = d1.compareTo(d2); if (cassandraCompare != javaCoreCompare) { System.err.println("[" + testNum + "]got incorrect result from LongType compare method." + "\n\tdate1: " + d1 + "\n\tdate2: " + d2 + "\n\tcassandraCompare: " + cassandraCompare + "\n\tjavaCoreCompare: " + javaCoreCompare ); } } } {code} If you will run the code you will se next output: {code} [5]got incorrect result from LongType compare method. date1: Wed Dec 31 23:58:20 GMT 1969 date2: Thu Jan 01 00:03:20 GMT 1970 cassandraCompare: 1 javaCoreCompare: -1 [6]got incorrect result from LongType compare method. date1: Thu Jan 01 00:03:20 GMT 1970 date2: Wed Dec 31 23:58:20 GMT 1969 cassandraCompare: -1 javaCoreCompare: 1 [7]got incorrect result from LongType compare method. date1: Thu Jan 01 00:01:40 GMT 1970 date2: Wed Dec 31 23:56:40 GMT 1969 cassandraCompare: -1 javaCoreCompare: 1 [8]got incorrect result from LongType compare method. date1: Wed Dec 31 23:56:40 GMT 1969 date2: Thu Jan 01 00:01:40 GMT 1970 cassandraCompare: 1 javaCoreCompare: -1 {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] cassandra git commit: Fix memory leak in Ref due to ConcurrentLinkedQueue behaviour
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 514dcd944 -> 6068efbaf Fix memory leak in Ref due to ConcurrentLinkedQueue behaviour patch by benedict; reviewed by marcus for CASSANDRA-9549 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7c5fc40b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7c5fc40b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7c5fc40b Branch: refs/heads/cassandra-2.2 Commit: 7c5fc40b8b644e05c32479f2581309f75f981421 Parents: 46cc577 Author: Benedict Elliott Smith Authored: Wed Jun 17 17:02:03 2015 +0100 Committer: Benedict Elliott Smith Committed: Wed Jun 17 17:02:03 2015 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/concurrent/Ref.java | 8 +++- .../cassandra/utils/concurrent/RefCountedTest.java | 15 +++ 3 files changed, 19 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 5f9187c..009d974 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.1.7 + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) Merged from 2.0 * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/src/java/org/apache/cassandra/utils/concurrent/Ref.java -- diff --git a/src/java/org/apache/cassandra/utils/concurrent/Ref.java b/src/java/org/apache/cassandra/utils/concurrent/Ref.java index 4e6cef7..f9876de 100644 --- a/src/java/org/apache/cassandra/utils/concurrent/Ref.java +++ b/src/java/org/apache/cassandra/utils/concurrent/Ref.java @@ -2,12 +2,10 @@ package org.apache.cassandra.utils.concurrent; import java.lang.ref.PhantomReference; import java.lang.ref.ReferenceQueue; +import java.util.Collection; import java.util.Collections; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ConcurrentLinkedQueue; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; +import java.util.concurrent.*; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; @@ -233,7 +231,7 @@ public final class Ref implements RefCounted, AutoCloseable { // we need to retain a reference to each of the PhantomReference instances // we are using to track individual refs -private final ConcurrentLinkedQueue locallyExtant = new ConcurrentLinkedQueue<>(); +private final Collection locallyExtant = new ConcurrentLinkedDeque<>(); // the number of live refs private final AtomicInteger counts = new AtomicInteger(); // the object to call to cleanup when our refs are all finished with http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java -- diff --git a/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java index a9247cd..bb173fe 100644 --- a/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java +++ b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java @@ -21,6 +21,7 @@ package org.apache.cassandra.utils.concurrent; import org.junit.Test; import junit.framework.Assert; +import org.apache.cassandra.utils.ObjectSizes; public class RefCountedTest { @@ -82,4 +83,18 @@ public class RefCountedTest { } } + +@Test +public void testMemoryLeak() +{ +Tidier tidier = new Tidier(); +Ref ref = new Ref(null, tidier); +long initialSize = ObjectSizes.measureDeep(ref); +for (int i = 0 ; i < 1000 ; i++) +ref.ref().release(); +long finalSize = ObjectSizes.measureDeep(ref); +if (finalSize > initialSize * 2) +throw new AssertionError(); +ref.release(); +} }
cassandra git commit: Fix memory leak in Ref due to ConcurrentLinkedQueue behaviour
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 46cc577a8 -> 7c5fc40b8 Fix memory leak in Ref due to ConcurrentLinkedQueue behaviour patch by benedict; reviewed by marcus for CASSANDRA-9549 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7c5fc40b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7c5fc40b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7c5fc40b Branch: refs/heads/cassandra-2.1 Commit: 7c5fc40b8b644e05c32479f2581309f75f981421 Parents: 46cc577 Author: Benedict Elliott Smith Authored: Wed Jun 17 17:02:03 2015 +0100 Committer: Benedict Elliott Smith Committed: Wed Jun 17 17:02:03 2015 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/concurrent/Ref.java | 8 +++- .../cassandra/utils/concurrent/RefCountedTest.java | 15 +++ 3 files changed, 19 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 5f9187c..009d974 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.1.7 + * Fix memory leak in Ref due to ConcurrentLinkedQueue.remove() behaviour (CASSANDRA-9549) Merged from 2.0 * Periodically submit background compaction tasks (CASSANDRA-9592) * Set HAS_MORE_PAGES flag to false when PagingState is null (CASSANDRA-9571) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/src/java/org/apache/cassandra/utils/concurrent/Ref.java -- diff --git a/src/java/org/apache/cassandra/utils/concurrent/Ref.java b/src/java/org/apache/cassandra/utils/concurrent/Ref.java index 4e6cef7..f9876de 100644 --- a/src/java/org/apache/cassandra/utils/concurrent/Ref.java +++ b/src/java/org/apache/cassandra/utils/concurrent/Ref.java @@ -2,12 +2,10 @@ package org.apache.cassandra.utils.concurrent; import java.lang.ref.PhantomReference; import java.lang.ref.ReferenceQueue; +import java.util.Collection; import java.util.Collections; import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.ConcurrentLinkedQueue; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; +import java.util.concurrent.*; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; @@ -233,7 +231,7 @@ public final class Ref implements RefCounted, AutoCloseable { // we need to retain a reference to each of the PhantomReference instances // we are using to track individual refs -private final ConcurrentLinkedQueue locallyExtant = new ConcurrentLinkedQueue<>(); +private final Collection locallyExtant = new ConcurrentLinkedDeque<>(); // the number of live refs private final AtomicInteger counts = new AtomicInteger(); // the object to call to cleanup when our refs are all finished with http://git-wip-us.apache.org/repos/asf/cassandra/blob/7c5fc40b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java -- diff --git a/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java index a9247cd..bb173fe 100644 --- a/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java +++ b/test/unit/org/apache/cassandra/utils/concurrent/RefCountedTest.java @@ -21,6 +21,7 @@ package org.apache.cassandra.utils.concurrent; import org.junit.Test; import junit.framework.Assert; +import org.apache.cassandra.utils.ObjectSizes; public class RefCountedTest { @@ -82,4 +83,18 @@ public class RefCountedTest { } } + +@Test +public void testMemoryLeak() +{ +Tidier tidier = new Tidier(); +Ref ref = new Ref(null, tidier); +long initialSize = ObjectSizes.measureDeep(ref); +for (int i = 0 ; i < 1000 ; i++) +ref.ref().release(); +long finalSize = ObjectSizes.measureDeep(ref); +if (finalSize > initialSize * 2) +throw new AssertionError(); +ref.release(); +} }
[jira] [Commented] (CASSANDRA-9499) Introduce writeVInt method to DataOutputStreamPlus
[ https://issues.apache.org/jira/browse/CASSANDRA-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589935#comment-14589935 ] Benedict commented on CASSANDRA-9499: - bq. Doesn't shifting the continuation bits to the first byte penalize the range of 1-byte values? No. You have to have a single continuation "bit" for either encoding. They're literally equivalent, just with their bit positions swapped around, except in the single byte case, in which case they are exactly identical. Run-length encoding means you count the number of set (or unset - this would actually be cleaner) contiguously at the top; the first that isn't (or is) tells you how many more bytes you need to read. i.e. if the first bit is unset, you're done, and the remaining 7 bits are value. If all 8 bits are set, we need to read a full long. This encoding gives us pretty ideal behaviour, of cheap operation over the single byte representation, followed by consistent behaviour across all the remaining possible values. It also lets us easily avoid wasting a whole byte for the final bit of a long. This is equivalent to Aleksey's characterisation. > Introduce writeVInt method to DataOutputStreamPlus > -- > > Key: CASSANDRA-9499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9499 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.0 beta 1 > > > CASSANDRA-8099 really could do with a writeVInt method, for both fixing > CASSANDRA-9498 but also efficiently encoding timestamp/deletion deltas. It > should be possible to make an especially efficient implementation against > BufferedDataOutputStreamPlus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9499) Introduce writeVInt method to DataOutputStreamPlus
[ https://issues.apache.org/jira/browse/CASSANDRA-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589920#comment-14589920 ] Aleksey Yeschenko commented on CASSANDRA-9499: -- I was thinking that we'd just implement what UTF8 does, except without self-synchronization semantics, which we don't really care about (so using the full range of the subsequent bytes, instead of using up their first 2 bits with the '10' marker). They way you tell the difference between length bits and value bits in the first byte is by separating the length bits and value bits by a mandatory 0 bit. > Introduce writeVInt method to DataOutputStreamPlus > -- > > Key: CASSANDRA-9499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9499 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.0 beta 1 > > > CASSANDRA-8099 really could do with a writeVInt method, for both fixing > CASSANDRA-9498 but also efficiently encoding timestamp/deletion deltas. It > should be possible to make an especially efficient implementation against > BufferedDataOutputStreamPlus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9499) Introduce writeVInt method to DataOutputStreamPlus
[ https://issues.apache.org/jira/browse/CASSANDRA-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589904#comment-14589904 ] Ariel Weisberg commented on CASSANDRA-9499: --- Doesn't shifting the continuation bits to the first byte penalize the range of 1-byte values? Supposedly that is the most common and so having 1 or 2 byte values with as much range as possible is a big advantage. Comments in the Google code make it seem like it's all about 1-byte values and that is the thing to optimize for. In practice if you only have to process one or two bytes the current implementation won't have to loop that many times. I don't understand how you can use a variable number of top order bits in the first bytes to indicate the number of bytes. How can you tell the difference between length bytes and value bytes? The way the existing code that doesn't use length extension works (and is simple) is that it drops the remaining bits of the first byte. That is something we want to get away from I thought. > Introduce writeVInt method to DataOutputStreamPlus > -- > > Key: CASSANDRA-9499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9499 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.0 beta 1 > > > CASSANDRA-8099 really could do with a writeVInt method, for both fixing > CASSANDRA-9498 but also efficiently encoding timestamp/deletion deltas. It > should be possible to make an especially efficient implementation against > BufferedDataOutputStreamPlus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9583) test-compression could run multiple unit tests in parallel like test
[ https://issues.apache.org/jira/browse/CASSANDRA-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589889#comment-14589889 ] Ariel Weisberg commented on CASSANDRA-9583: --- The tests passed. If you +1 the last commit we will be in business. > test-compression could run multiple unit tests in parallel like test > > > Key: CASSANDRA-9583 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9583 > Project: Cassandra > Issue Type: Test >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9563) Rename class for DATE type in Java driver
[ https://issues.apache.org/jira/browse/CASSANDRA-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589887#comment-14589887 ] Robert Stupp commented on CASSANDRA-9563: - Just give me a hint, when it's merged into driver's 2.2 branch. I can quickly build a snapshot of the driver and integrate it into the branch/patch. > Rename class for DATE type in Java driver > - > > Key: CASSANDRA-9563 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9563 > Project: Cassandra > Issue Type: Improvement >Reporter: Olivier Michallat >Assignee: Robert Stupp >Priority: Minor > Fix For: 2.2.0 rc2 > > > An early preview of the Java driver 2.2 was provided for inclusion in > Cassandra 2.2.0-rc1. It uses a custom Java type to represent CQL type > {{DATE}}. Currently that Java type is called {{DateWithoutTime}}. > We'd like to rename it to {{LocalDate}}. This would be a breaking change for > Cassandra, because that type is visible from UDF implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9563) Rename class for DATE type in Java driver
[ https://issues.apache.org/jira/browse/CASSANDRA-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Stupp updated CASSANDRA-9563: Fix Version/s: (was: 2.2.x) 2.2.0 rc2 > Rename class for DATE type in Java driver > - > > Key: CASSANDRA-9563 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9563 > Project: Cassandra > Issue Type: Improvement >Reporter: Olivier Michallat >Assignee: Robert Stupp >Priority: Minor > Fix For: 2.2.0 rc2 > > > An early preview of the Java driver 2.2 was provided for inclusion in > Cassandra 2.2.0-rc1. It uses a custom Java type to represent CQL type > {{DATE}}. Currently that Java type is called {{DateWithoutTime}}. > We'd like to rename it to {{LocalDate}}. This would be a breaking change for > Cassandra, because that type is visible from UDF implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9608) Support Java 9
Robert Stupp created CASSANDRA-9608: --- Summary: Support Java 9 Key: CASSANDRA-9608 URL: https://issues.apache.org/jira/browse/CASSANDRA-9608 Project: Cassandra Issue Type: Task Reporter: Robert Stupp Priority: Minor This ticket is intended to group all issues found to support Java 9 in the future. >From what I've found out so far: * Maven dependency {{com.sun:tools:jar:0}} via cobertura cannot be resolved. It can be easily solved using this patch: {code} - + + + {code} * Another issue is that {{sun.misc.Unsafe}} no longer contains the methods {{monitorEnter}} + {{monitorExit}}. These methods are used by {{o.a.c.utils.concurrent.Locks}} which is only used by {{o.a.c.db.AtomicBTreeColumns}}. I don't mind to start working on this yet since Java 9 is in a too early development phase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9503) Update CQL doc reflecting current keywords
[ https://issues.apache.org/jira/browse/CASSANDRA-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589871#comment-14589871 ] Tyler Hobbs commented on CASSANDRA-9503: bq. Do we need to update that directly, or is it part of the release process? Once we commit the changes to the textile file, I can regenerate the site documentation. It's in a separate ASF repository. > Update CQL doc reflecting current keywords > -- > > Key: CASSANDRA-9503 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9503 > Project: Cassandra > Issue Type: Bug > Components: Documentation & website >Reporter: Adam Holmberg >Assignee: Adam Holmberg >Priority: Trivial > Fix For: 2.2.x > > Attachments: cql_keywords.txt > > > The table in doc/cql3/CQL.textile#appendixA is outdated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9503) Update CQL doc reflecting current keywords
[ https://issues.apache.org/jira/browse/CASSANDRA-9503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Hobbs updated CASSANDRA-9503: --- Reviewer: Benjamin Lerer (was: Adam Holmberg) Assignee: Adam Holmberg (was: Tyler Hobbs) Assigning to [~blerer] for review. > Update CQL doc reflecting current keywords > -- > > Key: CASSANDRA-9503 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9503 > Project: Cassandra > Issue Type: Bug > Components: Documentation & website >Reporter: Adam Holmberg >Assignee: Adam Holmberg >Priority: Trivial > Fix For: 2.2.x > > Attachments: cql_keywords.txt > > > The table in doc/cql3/CQL.textile#appendixA is outdated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9460) NullPointerException Creating Digest
[ https://issues.apache.org/jira/browse/CASSANDRA-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Hobbs updated CASSANDRA-9460: --- Reviewer: Benjamin Lerer (was: Tyler Hobbs) Assigning [~blerer] to review. bq. Do you think we need additional dtests or is consistency_test.py enough? I think consistency_test.py is probably good enough, but since we know there can be a problem with race conditions, it might be good to make the test loop a few times instead of doing a single pass. > NullPointerException Creating Digest > > > Key: CASSANDRA-9460 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9460 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Tyler Hobbs >Assignee: Stefania > Fix For: 2.1.x > > Attachments: node2.log > > > In the {{consistency_test.TestConsistency.short_read_test}} dtest against > cassandra-2.1, the following error occured: > {noformat} > ERROR [ReadRepairStage:3] 2015-05-22 16:35:25,034 CassandraDaemon.java:223 - > Exception in thread Thread[ReadRepairStage:3,5,main] > java.lang.NullPointerException: null > at > org.apache.cassandra.db.ColumnFamily.updateDigest(ColumnFamily.java:390) > ~[main/:na] > at org.apache.cassandra.db.ColumnFamily.digest(ColumnFamily.java:383) > ~[main/:na] > at > org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:84) > ~[main/:na] > at > org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:28) > ~[main/:na] > at > org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:176) > ~[main/:na] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_80] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0_80] > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] > {noformat} > From a glance at the code in the stacktrace, it looks like there was a null > cell in the ColumnFamily that we were creating a digest of. This error is > probably particular to "short reads". > Here's the failing test: > http://cassci.datastax.com/job/cassandra-2.1_dtest/lastCompletedBuild/testReport/consistency_test/TestConsistency/short_read_test/. > I've attached the logs for the node with the error. > We saw this issue against 2.1, but the problem may also exist with 2.0 and/or > 2.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9563) Rename class for DATE type in Java driver
[ https://issues.apache.org/jira/browse/CASSANDRA-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589804#comment-14589804 ] Olivier Michallat commented on CASSANDRA-9563: -- We have a patch ready for JAVA-810, let us know when you need a new version of the driver. I agree that this has to go in 2.2rc2, could you change the fix version? > Rename class for DATE type in Java driver > - > > Key: CASSANDRA-9563 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9563 > Project: Cassandra > Issue Type: Improvement >Reporter: Olivier Michallat >Assignee: Robert Stupp >Priority: Minor > Fix For: 2.2.x > > > An early preview of the Java driver 2.2 was provided for inclusion in > Cassandra 2.2.0-rc1. It uses a custom Java type to represent CQL type > {{DATE}}. Currently that Java type is called {{DateWithoutTime}}. > We'd like to rename it to {{LocalDate}}. This would be a breaking change for > Cassandra, because that type is visible from UDF implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9532) Provide access to select statement's real column definitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589790#comment-14589790 ] Benjamin Lerer commented on CASSANDRA-9532: --- In my opinion, we should encapsulate the column definitions as part of {{SelectionColumnMapping}}. It makes sense to keep them together and will simplify the {{Selection}} code. In 2.2 and trunk we will then be able to populate {{SelectionColumnMapping}} via {{SelectorFactories.createFactoriesAndCollectColumnDefinitions}}. For the {{SelectStatement.getSelection()}} and {{SelectStatement.getRestrictions}} the javadoc should specify why they are public. I have the bad habit to remove unused methods. I think that the method {{SelectionColumnMapping.emptyMapping()}} has a confusing name. I expected the method to create an immutable empty {{SelectionColumnMapping}} (like Collections.emptyList()). > Provide access to select statement's real column definitions > > > Key: CASSANDRA-9532 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9532 > Project: Cassandra > Issue Type: Improvement >Reporter: mck >Assignee: Sam Tunnicliffe > Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x > > Attachments: 9532-2.0-v2.txt, 9532-2.1-v2.txt, 9532-2.2-v2.txt, > 9532-trunk-v2.txt, cassandra-2.0-9532.txt, cassandra-2.1-9532.txt, > cassandra-2.2-9532.txt, trunk-9532.txt > > > Currently there is no way to get access to the real ColumnDefinitions being > used in a SelectStatement. > This information is there in > {{selectStatement.selection.columns}} but is private. > Giving public access would make it possible for third-party implementations > of a {{QueryHandler}} to work accurately with the real columns being queried > and not have to work-around column aliases (or when the rawSelectors don't > map directly to ColumnDefinitions, eg in Selection.fromSelectors(..), like > functions), which is what one has to do today with going through > ResultSet.metadata.names. > This issue provides a very minimal patch to provide access to the already > final and immutable fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8099) Refactor and modernize the storage engine
[ https://issues.apache.org/jira/browse/CASSANDRA-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589744#comment-14589744 ] Branimir Lambov commented on CASSANDRA-8099: bq. it's not true that "an open marker is immediately followed by the corresponding close marker", there can be some rows between an open and a close marker. However, the guarantee that iterators should provide is that those rows between an open and close marker are not deleted by the range tombstone. It would be great to have this clarification in the doc. This is a point that I was missing; I will post a new version of the test that takes this into account. > Refactor and modernize the storage engine > - > > Key: CASSANDRA-8099 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8099 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Fix For: 3.0 beta 1 > > Attachments: 8099-nit > > > The current storage engine (which for this ticket I'll loosely define as "the > code implementing the read/write path") is suffering from old age. One of the > main problem is that the only structure it deals with is the cell, which > completely ignores the more high level CQL structure that groups cell into > (CQL) rows. > This leads to many inefficiencies, like the fact that during a reads we have > to group cells multiple times (to count on replica, then to count on the > coordinator, then to produce the CQL resultset) because we forget about the > grouping right away each time (so lots of useless cell names comparisons in > particular). But outside inefficiencies, having to manually recreate the CQL > structure every time we need it for something is hindering new features and > makes the code more complex that it should be. > Said storage engine also has tons of technical debt. To pick an example, the > fact that during range queries we update {{SliceQueryFilter.count}} is pretty > hacky and error prone. Or the overly complex ways {{AbstractQueryPager}} has > to go into to simply "remove the last query result". > So I want to bite the bullet and modernize this storage engine. I propose to > do 2 main things: > # Make the storage engine more aware of the CQL structure. In practice, > instead of having partitions be a simple iterable map of cells, it should be > an iterable list of row (each being itself composed of per-column cells, > though obviously not exactly the same kind of cell we have today). > # Make the engine more iterative. What I mean here is that in the read path, > we end up reading all cells in memory (we put them in a ColumnFamily object), > but there is really no reason to. If instead we were working with iterators > all the way through, we could get to a point where we're basically > transferring data from disk to the network, and we should be able to reduce > GC substantially. > Please note that such refactor should provide some performance improvements > right off the bat but it's not it's primary goal either. It's primary goal is > to simplify the storage engine and adds abstraction that are better suited to > further optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9606) this query is not supported in new version
[ https://issues.apache.org/jira/browse/CASSANDRA-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589723#comment-14589723 ] Benjamin Lerer edited comment on CASSANDRA-9606 at 6/17/15 12:55 PM: - The query should be invalid in 2.0. Cassandra does not allow multi-column restrictions and single-column restrictions on the same column. What it allow is: {{select * from test where a=1 and (b)<(6) and (b,c) > (3,2) limit 2;}}. Which will give you exactly what you want. Be aware that mixing multi and single column restrictions has only been properly fixed in 2.0.15 and 2.1.5 by CASSANDRA-8613. If what you want to do is paging, I advise you to check if the latest version of the driver that you use is not supporting manual paging like the java driver: http://datastax.github.io/java-driver/2.0.10/features/paging/#manual-paging was (Author: blerer): The query should be invalid in 2.0. We do not allow multi-column restrictions and single-column restrictions on the same column. What we allow is: {{select * from test where a=1 and (b)<(6) and (b,c) > (3,2) limit 2;}}. Which will give you exactly what you want. Be aware that mixing multi and single column restrictions has only been properly fixed in 2.0.15 and 2.1.5 by CASSANDRA-8613. If what you want to do is paging, I advise you to check if the latest version of the driver that you use is not supporting manual paging like the java driver: http://datastax.github.io/java-driver/2.0.10/features/paging/#manual-paging > this query is not supported in new version > -- > > Key: CASSANDRA-9606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9606 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: cassandra 2.1.6 > jdk 1.7.0_55 >Reporter: zhaoyan >Assignee: Benjamin Lerer > > Background: > 1、create a table: > {code} > CREATE TABLE test ( > a int, > b int, > c int, > d int, > PRIMARY KEY (a, b, c) > ); > {code} > 2、query by a=1 and b<6 > {code} > select * from test where a=1 and b<6; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (6 rows) > {code} > 3、query by page > first page: > {code} > select * from test where a=1 and b<6 limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > (2 rows) > {code} > second page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > (2 rows) > {code} > last page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,5) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (2 rows) > {code} > question: > this query by page is ok when cassandra 2.0.8. > but is not supported in the latest version 2.1.6 > when execute: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > {code} > get one error message: > InvalidRequest: code=2200 [Invalid query] message="Column "b" cannot have > both tuple-notation inequalities and single-column inequalities: (b, c) > (3, > 2)" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9606) this query is not supported in new version
[ https://issues.apache.org/jira/browse/CASSANDRA-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589727#comment-14589727 ] Sylvain Lebresne commented on CASSANDRA-9606: - {{(b)<(6)}} should be absolutely equivalent {{b<6}} or it's a bug. > this query is not supported in new version > -- > > Key: CASSANDRA-9606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9606 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: cassandra 2.1.6 > jdk 1.7.0_55 >Reporter: zhaoyan >Assignee: Benjamin Lerer > > Background: > 1、create a table: > {code} > CREATE TABLE test ( > a int, > b int, > c int, > d int, > PRIMARY KEY (a, b, c) > ); > {code} > 2、query by a=1 and b<6 > {code} > select * from test where a=1 and b<6; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (6 rows) > {code} > 3、query by page > first page: > {code} > select * from test where a=1 and b<6 limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > (2 rows) > {code} > second page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > (2 rows) > {code} > last page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,5) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (2 rows) > {code} > question: > this query by page is ok when cassandra 2.0.8. > but is not supported in the latest version 2.1.6 > when execute: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > {code} > get one error message: > InvalidRequest: code=2200 [Invalid query] message="Column "b" cannot have > both tuple-notation inequalities and single-column inequalities: (b, c) > (3, > 2)" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9606) this query is not supported in new version
[ https://issues.apache.org/jira/browse/CASSANDRA-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589723#comment-14589723 ] Benjamin Lerer commented on CASSANDRA-9606: --- The query should be invalid in 2.0. We do not allow multi-column restrictions and single-column restrictions on the same column. What we allow is: {{select * from test where a=1 and (b)<(6) and (b,c) > (3,2) limit 2;}}. Which will give you exactly what you want. Be aware that mixing multi and single column restrictions has only been properly fixed in 2.0.15 and 2.1.5 by CASSANDRA-8613. If what you want to do is paging, I advise you to check if the latest version of the driver that you use is not supporting manual paging like the java driver: http://datastax.github.io/java-driver/2.0.10/features/paging/#manual-paging > this query is not supported in new version > -- > > Key: CASSANDRA-9606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9606 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: cassandra 2.1.6 > jdk 1.7.0_55 >Reporter: zhaoyan >Assignee: Benjamin Lerer > > Background: > 1、create a table: > {code} > CREATE TABLE test ( > a int, > b int, > c int, > d int, > PRIMARY KEY (a, b, c) > ); > {code} > 2、query by a=1 and b<6 > {code} > select * from test where a=1 and b<6; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (6 rows) > {code} > 3、query by page > first page: > {code} > select * from test where a=1 and b<6 limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > (2 rows) > {code} > second page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > (2 rows) > {code} > last page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,5) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (2 rows) > {code} > question: > this query by page is ok when cassandra 2.0.8. > but is not supported in the latest version 2.1.6 > when execute: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > {code} > get one error message: > InvalidRequest: code=2200 [Invalid query] message="Column "b" cannot have > both tuple-notation inequalities and single-column inequalities: (b, c) > (3, > 2)" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8099) Refactor and modernize the storage engine
[ https://issues.apache.org/jira/browse/CASSANDRA-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589721#comment-14589721 ] Sylvain Lebresne commented on CASSANDRA-8099: - I'll look more closely at your test and fix any brokenness: it does seem the results are not what they are supposed to be. For the record however, I'll note that it's not true that "an open marker is immediately followed by the corresponding close marker", there can be some rows between an open and a close marker. However, the guarantee that iterators should provide is that those rows between an open and close marker are not deleted by the range tombstone (this doesn't make the tests result above any more right, but wanted to clarify). > Refactor and modernize the storage engine > - > > Key: CASSANDRA-8099 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8099 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Fix For: 3.0 beta 1 > > Attachments: 8099-nit > > > The current storage engine (which for this ticket I'll loosely define as "the > code implementing the read/write path") is suffering from old age. One of the > main problem is that the only structure it deals with is the cell, which > completely ignores the more high level CQL structure that groups cell into > (CQL) rows. > This leads to many inefficiencies, like the fact that during a reads we have > to group cells multiple times (to count on replica, then to count on the > coordinator, then to produce the CQL resultset) because we forget about the > grouping right away each time (so lots of useless cell names comparisons in > particular). But outside inefficiencies, having to manually recreate the CQL > structure every time we need it for something is hindering new features and > makes the code more complex that it should be. > Said storage engine also has tons of technical debt. To pick an example, the > fact that during range queries we update {{SliceQueryFilter.count}} is pretty > hacky and error prone. Or the overly complex ways {{AbstractQueryPager}} has > to go into to simply "remove the last query result". > So I want to bite the bullet and modernize this storage engine. I propose to > do 2 main things: > # Make the storage engine more aware of the CQL structure. In practice, > instead of having partitions be a simple iterable map of cells, it should be > an iterable list of row (each being itself composed of per-column cells, > though obviously not exactly the same kind of cell we have today). > # Make the engine more iterative. What I mean here is that in the read path, > we end up reading all cells in memory (we put them in a ColumnFamily object), > but there is really no reason to. If instead we were working with iterators > all the way through, we could get to a point where we're basically > transferring data from disk to the network, and we should be able to reduce > GC substantially. > Please note that such refactor should provide some performance improvements > right off the bat but it's not it's primary goal either. It's primary goal is > to simplify the storage engine and adds abstraction that are better suited to > further optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-9606) this query is not supported in new version
[ https://issues.apache.org/jira/browse/CASSANDRA-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer reassigned CASSANDRA-9606: - Assignee: Benjamin Lerer > this query is not supported in new version > -- > > Key: CASSANDRA-9606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9606 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: cassandra 2.1.6 > jdk 1.7.0_55 >Reporter: zhaoyan >Assignee: Benjamin Lerer > > Background: > 1、create a table: > {code} > CREATE TABLE test ( > a int, > b int, > c int, > d int, > PRIMARY KEY (a, b, c) > ); > {code} > 2、query by a=1 and b<6 > {code} > select * from test where a=1 and b<6; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (6 rows) > {code} > 3、query by page > first page: > {code} > select * from test where a=1 and b<6 limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 1 | 2 > 1 | 3 | 2 | 2 > (2 rows) > {code} > second page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 3 | 4 | 2 > 1 | 3 | 5 | 2 > (2 rows) > {code} > last page: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,5) limit 2; > a | b | c | d > ---+---+---+--- > 1 | 4 | 4 | 2 > 1 | 5 | 5 | 2 > (2 rows) > {code} > question: > this query by page is ok when cassandra 2.0.8. > but is not supported in the latest version 2.1.6 > when execute: > {code} > select * from test where a=1 and b<6 and (b,c) > (3,2) limit 2; > {code} > get one error message: > InvalidRequest: code=2200 [Invalid query] message="Column "b" cannot have > both tuple-notation inequalities and single-column inequalities: (b, c) > (3, > 2)" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9549) Memory leak
[ https://issues.apache.org/jira/browse/CASSANDRA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589683#comment-14589683 ] Marcus Eriksson commented on CASSANDRA-9549: +1 > Memory leak > > > Key: CASSANDRA-9549 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9549 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.5. 9 node cluster in EC2 (m1.large nodes, > 2 cores 7.5G memory, 800G platter for cassandra data, root partition and > commit log are on SSD EBS with sufficient IOPS), 3 nodes/availablity zone, 1 > replica/zone > JVM: /usr/java/jdk1.8.0_40/jre/bin/java > JVM Flags besides CP: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar > -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms2G -Xmx2G -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler > -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled > -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark > -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.rmi.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra > -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid > Kernel: Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux >Reporter: Ivar Thorson >Assignee: Benedict >Priority: Critical > Fix For: 2.1.x > > Attachments: c4_system.log, c7fromboot.zip, cassandra.yaml, > cpu-load.png, memoryuse.png, ref-java-errors.jpeg, suspect.png, two-loads.png > > > We have been experiencing a severe memory leak with Cassandra 2.1.5 that, > over the period of a couple of days, eventually consumes all of the available > JVM heap space, putting the JVM into GC hell where it keeps trying CMS > collection but can't free up any heap space. This pattern happens for every > node in our cluster and is requiring rolling cassandra restarts just to keep > the cluster running. We have upgraded the cluster per Datastax docs from the > 2.0 branch a couple of months ago and have been using the data from this > cluster for more than a year without problem. > As the heap fills up with non-GC-able objects, the CPU/OS load average grows > along with it. Heap dumps reveal an increasing number of > java.util.concurrent.ConcurrentLinkedQueue$Node objects. We took heap dumps > over a 2 day period, and watched the number of Node objects go from 4M, to > 19M, to 36M, and eventually about 65M objects before the node stops > responding. The screen capture of our heap dump is from the 19M measurement. > Load on the cluster is minimal. We can see this effect even with only a > handful of writes per second. (See attachments for Opscenter snapshots during > very light loads and heavier loads). Even with only 5 reads a sec we see this > behavior. > Log files show repeated errors in Ref.java:181 and Ref.java:279 and "LEAK > detected" messages: > {code} > ERROR [CompactionExecutor:557] 2015-06-01 18:27:36,978 Ref.java:279 - Error > when closing class > org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1302301946:/data1/data/ourtablegoeshere-ka-1150 > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32680b31 > rejected from > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@573464d6[Terminated, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1644] > {code} > {code} > ERROR [Reference-Reaper:1] 2015-06-01 18:27:37,083 Ref.java:181 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@74b5df92) to class > org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@2054303604:/data2/data/ourtablegoeshere-ka-1151 > was not released before the reference was garbage collected > {code} > This might be related to [CASSANDRA-8723]? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8099) Refactor and modernize the storage engine
[ https://issues.apache.org/jira/browse/CASSANDRA-8099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589679#comment-14589679 ] Branimir Lambov commented on CASSANDRA-8099: The changes look good, I accept that clarifying the side effects and switching to Iterable is not something that needs to be taken care of as part of this ticket. On the range tombstones doc, I'm sorry, I totally failed to explain what I had in mind. I was looking for a statement in {{### Dealing with tombstones and shadowed cells}} that says an open marker is immediately followed by the corresponding close marker. This is a simple and easy to check statement which is equivalent to having both "does not shadow its own data" (pt 1) and "there is at most one active tombstone at any point" (pt 4). However, to clarify this I went and looked further into how tombstones work in the new code and after looking at it I do not think the merging code could work correctly. To make certain I wrote a [partition merge test|https://github.com/apache/cassandra/commit/36709f4f8f81125def076c91ac8dffae2fdf71b0]. The test results are very broken, for example: {code} Merging 6<=[34] <8 8 19<=[99] <=36 40 47 50<[56] <66 67 68 72<=[26] <=73 89<=[97] <97 2<=[66] <19 19 34<=[58] <=35 36 40<[94] <41 42<[26] <=48 55<=[35] <=57 58 83 88 5 8 19 31<=[49] <=31 37<=[70] <=44 46<[79] <55 65 72<[57] <85 92<[45] <93 93 results in 2<=[66] <19 19<=[99] <=36 37<=[70] 40<[94] 41<=[70] 41<=[70] 44<=[26] 44<=[26] 46<[79] 55<=[56] 55<=[56] <66 67 68 72<=[26] 72<[57] <85 88 89<=[97] <97 java.lang.AssertionError: 40<[94] follows another open marker 37<=[70] java.lang.AssertionError: Duplicate marker 41<=[70] java.lang.AssertionError: Deletion time mismatch for position 44 expected: but was: {code} {code} Merging 4 6 13<=[62] <=26 34<=[89] <=34 47 48<[99] <52 54<=[37] <=57 78<[6] <=83 85 88<=[34] <91 0 20 33<=[58] <=33 37<=[84] <40 52 57 77 77 88<[14] <91 92<[15] <96 1 8 31 41<[17] <43 62<=[25] <=67 85 87 88 92 97 results in 0 1 4 6 8 13<=[62] <=26 31 33<=[58] <=33 34<=[89] <=34 37<=[84] <40 41<[17] <43 47 48<[99] <52 52 54<=[37] <=57 62<=[25] <=67 77 77 78<[6] <=83 87 88<=[34] <91 <91 92 92<[15] <96 97 java.lang.AssertionError: <91 should be preceded by open marker, found <91 java.lang.AssertionError: Deletion time mismatch for open 88<=[34] and close <91 expected: but was: {code} (where {{x>\[z\]}} means tombstone open marker at {{x}} with time {{z}} and {{ Refactor and modernize the storage engine > - > > Key: CASSANDRA-8099 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8099 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Fix For: 3.0 beta 1 > > Attachments: 8099-nit > > > The current storage engine (which for this ticket I'll loosely define as "the > code implementing the read/write path") is suffering from old age. One of the > main problem is that the only structure it deals with is the cell, which > completely ignores the more high level CQL structure that groups cell into > (CQL) rows. > This leads to many inefficiencies, like the fact that during a reads we have > to group cells multiple times (to count on replica, then to count on the > coordinator, then to produce the CQL resultset) because we forget about the > grouping right away each time (so lots of useless cell names comparisons in > particular). But outside inefficiencies, having to manually recreate the CQL > structure every time we need it for something is hindering new features and > makes the code more complex that it should be. > Said storage engine also has tons of technical debt. To pick an example, the > fact that during range queries we update {{SliceQueryFilter.count}} is pretty > hacky and error prone. Or the overly complex ways {{AbstractQueryPager}} has > to go into to simply "remove the last query result". > So I want to bite the bullet and modernize this storage engine. I propose to > do 2 main things: > # Make the storage engine more aware of the CQL structure. In practice, > instead of having partitions be a simple iterable map of cells, it should be > an iterable list of row (each being itself composed of per-column cells, > though obviously not exactly the same kind of cell we have today). > # Make the engine more iterative. What I mean here is that in the read path, > we end up reading all cells in memory (we put them in a ColumnFamily object), > but there is really no reason to. If instead we were working with iterators > all the way through, we could get to a point where we're basically > transferring data from disk to the network, and we should be able to reduce > GC substantially. > Please note that such refactor should provide some performance imp
[jira] [Updated] (CASSANDRA-9549) Memory leak
[ https://issues.apache.org/jira/browse/CASSANDRA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-9549: --- Reviewer: Marcus Eriksson Reproduced In: 2.1.6, 2.1.5 (was: 2.1.5, 2.1.6) > Memory leak > > > Key: CASSANDRA-9549 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9549 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.5. 9 node cluster in EC2 (m1.large nodes, > 2 cores 7.5G memory, 800G platter for cassandra data, root partition and > commit log are on SSD EBS with sufficient IOPS), 3 nodes/availablity zone, 1 > replica/zone > JVM: /usr/java/jdk1.8.0_40/jre/bin/java > JVM Flags besides CP: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar > -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms2G -Xmx2G -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler > -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled > -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark > -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.rmi.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra > -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid > Kernel: Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux >Reporter: Ivar Thorson >Assignee: Benedict >Priority: Critical > Fix For: 2.1.x > > Attachments: c4_system.log, c7fromboot.zip, cassandra.yaml, > cpu-load.png, memoryuse.png, ref-java-errors.jpeg, suspect.png, two-loads.png > > > We have been experiencing a severe memory leak with Cassandra 2.1.5 that, > over the period of a couple of days, eventually consumes all of the available > JVM heap space, putting the JVM into GC hell where it keeps trying CMS > collection but can't free up any heap space. This pattern happens for every > node in our cluster and is requiring rolling cassandra restarts just to keep > the cluster running. We have upgraded the cluster per Datastax docs from the > 2.0 branch a couple of months ago and have been using the data from this > cluster for more than a year without problem. > As the heap fills up with non-GC-able objects, the CPU/OS load average grows > along with it. Heap dumps reveal an increasing number of > java.util.concurrent.ConcurrentLinkedQueue$Node objects. We took heap dumps > over a 2 day period, and watched the number of Node objects go from 4M, to > 19M, to 36M, and eventually about 65M objects before the node stops > responding. The screen capture of our heap dump is from the 19M measurement. > Load on the cluster is minimal. We can see this effect even with only a > handful of writes per second. (See attachments for Opscenter snapshots during > very light loads and heavier loads). Even with only 5 reads a sec we see this > behavior. > Log files show repeated errors in Ref.java:181 and Ref.java:279 and "LEAK > detected" messages: > {code} > ERROR [CompactionExecutor:557] 2015-06-01 18:27:36,978 Ref.java:279 - Error > when closing class > org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1302301946:/data1/data/ourtablegoeshere-ka-1150 > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32680b31 > rejected from > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@573464d6[Terminated, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1644] > {code} > {code} > ERROR [Reference-Reaper:1] 2015-06-01 18:27:37,083 Ref.java:181 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@74b5df92) to class > org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@2054303604:/data2/data/ourtablegoeshere-ka-1151 > was not released before the reference was garbage collected > {code} > This might be related to [CASSANDRA-8723]? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[ https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589635#comment-14589635 ] Andreas Schnitzerling edited comment on CASSANDRA-8535 at 6/17/15 11:12 AM: On Windows 7, files and directories cannot be renamed, if they're still open, so we need to close them before: {code:title=src/java/org/apache/cassandra/io/sstable/SSTableWriter.java} private Pair close(FinishType type, long repairedAt) Descriptor descriptor = this.descriptor; if (type.isFinal) { dataFile.writeFullChecksum(descriptor); writeMetadata(descriptor, metadataComponents); // save the table of components SSTable.appendTOC(descriptor, components); +++ dataFile.close(); descriptor = rename(descriptor, components); } {code} Can somebody check/test and maybe commit it? Thx. was (Author: andie78): On Windows 7, files and directories cannot be renamed, if they're still open, so we need to close them before: {code:title=\src\java\org\apache\cassandra\io\sstable\SSTableWriter.java} private Pair close(FinishType type, long repairedAt) Descriptor descriptor = this.descriptor; if (type.isFinal) { dataFile.writeFullChecksum(descriptor); writeMetadata(descriptor, metadataComponents); // save the table of components SSTable.appendTOC(descriptor, components); +++ dataFile.close(); descriptor = rename(descriptor, components); } {code} Can somebody check/test and maybe commit it? Thx. > java.lang.RuntimeException: Failed to rename XXX to YYY > --- > > Key: CASSANDRA-8535 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8535 > Project: Cassandra > Issue Type: Bug > Environment: Windows 2008 X64 >Reporter: Leonid Shalupov >Assignee: Joshua McKenzie > Labels: Windows > Fix For: 2.1.5 > > Attachments: 8535_v1.txt, 8535_v2.txt, 8535_v3.txt > > > {code} > java.lang.RuntimeException: Failed to rename > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db > to > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db > at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170) > ~[main/:na] > at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200) > ~[main/:na] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) > ~[main/:na] > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226) > ~[main/:na] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > Caused by: java.nio.file.FileSystemException: > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db > -> > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db: > The process ca
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[ https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589635#comment-14589635 ] Andreas Schnitzerling commented on CASSANDRA-8535: -- On Windows 7, files and directories cannot be renamed, if they're still open, so we need to close them before: {code:title=\src\java\org\apache\cassandra\io\sstable\SSTableWriter.java} private Pair close(FinishType type, long repairedAt) Descriptor descriptor = this.descriptor; if (type.isFinal) { dataFile.writeFullChecksum(descriptor); writeMetadata(descriptor, metadataComponents); // save the table of components SSTable.appendTOC(descriptor, components); +++ dataFile.close(); descriptor = rename(descriptor, components); } {code} Can somebody check/test and maybe commit it? Thx. > java.lang.RuntimeException: Failed to rename XXX to YYY > --- > > Key: CASSANDRA-8535 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8535 > Project: Cassandra > Issue Type: Bug > Environment: Windows 2008 X64 >Reporter: Leonid Shalupov >Assignee: Joshua McKenzie > Labels: Windows > Fix For: 2.1.5 > > Attachments: 8535_v1.txt, 8535_v2.txt, 8535_v3.txt > > > {code} > java.lang.RuntimeException: Failed to rename > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db > to > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db > at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170) > ~[main/:na] > at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200) > ~[main/:na] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) > ~[main/:na] > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226) > ~[main/:na] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > Caused by: java.nio.file.FileSystemException: > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db > -> > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db: > The process cannot access the file because it is being used by another > process. > at > sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) > ~[na:1.7.0_45] > at > sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) > ~[na:1.7.0_45] > at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) > ~[na:1.7.0_45] > at > sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) > ~[na:1.7.0_45] > at java.nio.file.Files.move(Files.java:1345) ~[na:1.7.0_45] > at > org.apache.cassandra.io.util.FileUtils.atomicMoveWithFallback(FileUtils.java:184) > ~[main/:na] > at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:166) > ~[main/:na] > .
[jira] [Comment Edited] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589589#comment-14589589 ] Study Hsueh edited comment on CASSANDRA-9607 at 6/17/15 10:27 AM: -- This issues cause all of nodes downs. I will attach log later after I downgrade to 2.1.3... was (Author: study): This issues cause all of nodes downs. > Get high load after upgrading from 2.1.3 to cassandra 2.1.6 > --- > > Key: CASSANDRA-9607 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 > Project: Cassandra > Issue Type: Bug > Environment: OS: > CentOS 6 * 4 > Ubuntu 14.04 * 2 > JDK: Oracle JDK 7 >Reporter: Study Hsueh >Priority: Critical > Attachments: load.png > > > After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my > cassandra cluster grows from 0.x~1.x to 3.x~6.x. > What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
[ https://issues.apache.org/jira/browse/CASSANDRA-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589589#comment-14589589 ] Study Hsueh commented on CASSANDRA-9607: This issues cause all of nodes downs. > Get high load after upgrading from 2.1.3 to cassandra 2.1.6 > --- > > Key: CASSANDRA-9607 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 > Project: Cassandra > Issue Type: Bug > Environment: OS: > CentOS 6 * 4 > Ubuntu 14.04 * 2 > JDK: Oracle JDK 7 >Reporter: Study Hsueh >Priority: Critical > Attachments: load.png > > > After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my > cassandra cluster grows from 0.x~1.x to 3.x~6.x. > What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[ https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589583#comment-14589583 ] Maxim Podkolzine commented on CASSANDRA-8535: - I'm afraid this issue has to be reopened. I've got this in 2.1.6: 2015-06-15T17:31:44,797 [CompactionExecutor:1] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[CompactionExecutor:1,1,main] java.lang.RuntimeException: Failed to rename E:\Upsource\Upsource_2.0.262_new_cassandra\data\cassandra\data\upsource\content-f13ce210136211e59a87137398728adc\upsource-content-tmp-ka-18-Index.db to E:\Upsource\Upsource_2.0.262_new_cassandra\ data\cassandra\data\upsource\content-f13ce210136211e59a87137398728adc\upsource-content-ka-18-Index.db at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170) ~[apache-cassandra-2.1.6.jar:2.1.6] at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154) ~[apache-cassandra-2.1.6.jar:2.1.6] at org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:541) ~[apache-cassandra-2.1.6.jar:2.1.6] at org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:533) ~[apache-cassandra-2.1.6.jar:2.1.6] Can provide the full stacktrace if needed, but it looks the same as in the issue: Caused by: java.nio.file.FileSystemException: E:\Upsource\Upsource_2.0.262_new_cassandra\data\cassandra\data\upsource\uniqueidhistory_t-eef64d70136211e59a87137398728adc\upsource-uniqueidhistory_t-tmplink-ka-23-Index.db: The process cannot access the file because it is being used by another process. I can add that we've been running Cassandra build on 879691dd revision and it worked fine on Windows under same load. So it looks like one of the last changes in 2.1 branch broke it again. > java.lang.RuntimeException: Failed to rename XXX to YYY > --- > > Key: CASSANDRA-8535 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8535 > Project: Cassandra > Issue Type: Bug > Environment: Windows 2008 X64 >Reporter: Leonid Shalupov >Assignee: Joshua McKenzie > Labels: Windows > Fix For: 2.1.5 > > Attachments: 8535_v1.txt, 8535_v2.txt, 8535_v3.txt > > > {code} > java.lang.RuntimeException: Failed to rename > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db > to > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db > at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170) > ~[main/:na] > at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324) > ~[main/:na] > at > org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200) > ~[main/:na] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) > ~[main/:na] > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > ~[main/:na] > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226) > ~[main/:na] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_45] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > Caused by: java.nio.file.FileSystemException: > build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
[jira] [Commented] (CASSANDRA-7020) Incorrect result of query WHERE token(key) < -9223372036854775808 when using Murmur3Partitioner
[ https://issues.apache.org/jira/browse/CASSANDRA-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589575#comment-14589575 ] Anton commented on CASSANDRA-7020: -- The same when equal. {code} cqlsh:test1> select * from test where token("key") = -9223372036854775808; key | value -+-- 5 | ee 10 |j 1 | 8 | 2 | bbb 4 | dd 7 | 6 | fff 9 | 3 |c {code} Too bad. > Incorrect result of query WHERE token(key) < -9223372036854775808 when using > Murmur3Partitioner > --- > > Key: CASSANDRA-7020 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7020 > Project: Cassandra > Issue Type: Bug > Environment: cassandra 2.0.6-snapshot >Reporter: Piotr Kołaczkowski >Assignee: Marko Denda > > {noformat} > cqlsh:test1> select * from test where token("key") < -9223372036854775807; > (0 rows) > cqlsh:test1> select * from test where token("key") < -9223372036854775808; > key | value > -+-- >5 | ee > 10 |j >1 | >8 | >2 | bbb >4 | dd >7 | >6 | fff >9 | >3 |c > {noformat} > Expected: empty result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9549) Memory leak
[ https://issues.apache.org/jira/browse/CASSANDRA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589537#comment-14589537 ] Benedict commented on CASSANDRA-9549: - I've added a regression test to the branch. Could I get a reviewer please, and can we ship this soon? > Memory leak > > > Key: CASSANDRA-9549 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9549 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.5. 9 node cluster in EC2 (m1.large nodes, > 2 cores 7.5G memory, 800G platter for cassandra data, root partition and > commit log are on SSD EBS with sufficient IOPS), 3 nodes/availablity zone, 1 > replica/zone > JVM: /usr/java/jdk1.8.0_40/jre/bin/java > JVM Flags besides CP: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar > -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms2G -Xmx2G -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly > -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler > -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled > -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark > -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.rmi.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra > -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid > Kernel: Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux >Reporter: Ivar Thorson >Assignee: Benedict >Priority: Critical > Fix For: 2.1.x > > Attachments: c4_system.log, c7fromboot.zip, cassandra.yaml, > cpu-load.png, memoryuse.png, ref-java-errors.jpeg, suspect.png, two-loads.png > > > We have been experiencing a severe memory leak with Cassandra 2.1.5 that, > over the period of a couple of days, eventually consumes all of the available > JVM heap space, putting the JVM into GC hell where it keeps trying CMS > collection but can't free up any heap space. This pattern happens for every > node in our cluster and is requiring rolling cassandra restarts just to keep > the cluster running. We have upgraded the cluster per Datastax docs from the > 2.0 branch a couple of months ago and have been using the data from this > cluster for more than a year without problem. > As the heap fills up with non-GC-able objects, the CPU/OS load average grows > along with it. Heap dumps reveal an increasing number of > java.util.concurrent.ConcurrentLinkedQueue$Node objects. We took heap dumps > over a 2 day period, and watched the number of Node objects go from 4M, to > 19M, to 36M, and eventually about 65M objects before the node stops > responding. The screen capture of our heap dump is from the 19M measurement. > Load on the cluster is minimal. We can see this effect even with only a > handful of writes per second. (See attachments for Opscenter snapshots during > very light loads and heavier loads). Even with only 5 reads a sec we see this > behavior. > Log files show repeated errors in Ref.java:181 and Ref.java:279 and "LEAK > detected" messages: > {code} > ERROR [CompactionExecutor:557] 2015-06-01 18:27:36,978 Ref.java:279 - Error > when closing class > org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1302301946:/data1/data/ourtablegoeshere-ka-1150 > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32680b31 > rejected from > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@573464d6[Terminated, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1644] > {code} > {code} > ERROR [Reference-Reaper:1] 2015-06-01 18:27:37,083 Ref.java:181 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@74b5df92) to class > org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@2054303604:/data2/data/ourtablegoeshere-ka-1151 > was not released before the reference was garbage collected > {code} > This might be related to [CASSANDRA-8723]? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9607) Get high load after upgrading from 2.1.3 to cassandra 2.1.6
Study Hsueh created CASSANDRA-9607: -- Summary: Get high load after upgrading from 2.1.3 to cassandra 2.1.6 Key: CASSANDRA-9607 URL: https://issues.apache.org/jira/browse/CASSANDRA-9607 Project: Cassandra Issue Type: Bug Environment: OS: CentOS 6 * 4 Ubuntu 14.04 * 2 JDK: Oracle JDK 7 Reporter: Study Hsueh Priority: Critical Attachments: load.png After upgrading cassandra version from 2.1.3 to 2.1.6, the average load of my cassandra cluster grows from 0.x~1.x to 3.x~6.x. What kind of additional information should I provide for this problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5383) Windows 7 deleting/renaming files problem
[ https://issues.apache.org/jira/browse/CASSANDRA-5383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589504#comment-14589504 ] Andreas Schnitzerling commented on CASSANDRA-5383: -- I put it manually into the source: /src/org/apache/cassandra/io/util/RandomAccessReader.java Put (overwrite) the finalizer there. To avoid unnecessary logs after patching u can un-comment the warning (logger.error) in /sstable/SSTableDeletingTask.java To build u need maven (I use 3.0.5) and ant (1.9.2) and an internet-connection. I have to use ant on a dedicated server in front of the proxy since I couldn't get proxy-settings working in ant. After the modifications u can type "ant jar" if u just need the patched jar(s). "ant release" builds u the compressed bin-package. > Windows 7 deleting/renaming files problem > - > > Key: CASSANDRA-5383 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5383 > Project: Cassandra > Issue Type: Bug > Components: Tests >Affects Versions: 2.0 beta 1 >Reporter: Ryan McGuire >Assignee: Marcus Eriksson > Labels: qa-resolved > Fix For: 2.0.2 > > Attachments: > 0001-CASSANDRA-5383-cant-move-a-file-on-top-of-another-fi.patch, > 0001-CASSANDRA-5383-v2.patch, > 0001-use-Java7-apis-for-deleting-and-moving-files-and-cre.patch, > 5383-v3.patch, 5383_patch_v2_system.log, cant_move_file_patch.log, > test_log.5383.patch_v2.log.txt, v2+cant_move_file_patch.log > > > Two unit tests are failing on Windows 7 due to errors in renaming/deleting > files: > org.apache.cassandra.db.ColumnFamilyStoreTest: > {code} > [junit] Testsuite: org.apache.cassandra.db.ColumnFamilyStoreTest > [junit] Tests run: 27, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: > 13.904 sec > [junit] > [junit] - Standard Error - > [junit] ERROR 13:06:46,058 Unable to delete > build\test\cassandra\data\Keyspace1\Indexed2\Keyspace1-Indexed2.birthdate_index-ja-1-Data.db > (it will be removed on server restart; we'll also retry after GC) > [junit] ERROR 13:06:48,508 Fatal exception in thread > Thread[NonPeriodicTasks:1,5,main] > [junit] java.lang.RuntimeException: Tried to hard link to file that does > not exist > build\test\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-ja-7-Statistics.db > [junit] at > org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:72) > [junit] at > org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1057) > [junit] at > org.apache.cassandra.db.DataTracker$1.run(DataTracker.java:168) > [junit] at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > [junit] at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138) > [junit] at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) > [junit] at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) > [junit] at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > [junit] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > [junit] at java.lang.Thread.run(Thread.java:662) > [junit] - --- > [junit] Testcase: > testSliceByNamesCommandOldMetatada(org.apache.cassandra.db.ColumnFamilyStoreTest): > Caused an ERROR > [junit] Failed to rename > build\test\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-ja-6-Statistics.db-tmp > to > build\test\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-ja-6-Statistics.db > [junit] java.lang.RuntimeException: Failed to rename > build\test\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-ja-6-Statistics.db-tmp > to > build\test\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-ja-6-Statistics.db > [junit] at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:133) > [junit] at > org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:122) > [junit] at > org.apache.cassandra.db.compaction.LeveledManifest.mutateLevel(LeveledManifest.java:575) > [junit] at > org.apache.cassandra.db.ColumnFamilyStore.loadNewSSTables(ColumnFamilyStore.java:589) > [junit] at > org.apache.cassandra.db.ColumnFamilyStoreTest.testSliceByNamesCommandOldMetatada(ColumnFamilyStoreTest.java:885) > [junit] > [junit] > [junit] Testcase: > testRemoveUnifinishedCompactionLeftovers(org.apache.cassandra.db.ColumnFamilyStoreTest): > Caused an ER
[jira] [Comment Edited] (CASSANDRA-9499) Introduce writeVInt method to DataOutputStreamPlus
[ https://issues.apache.org/jira/browse/CASSANDRA-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589470#comment-14589470 ] Benedict edited comment on CASSANDRA-9499 at 6/17/15 8:47 AM: -- I'm confused as to why we need 10 bytes? Pretty much by definition a continuation bit encoding needs 9 bytes to represent 8 bytes. Let's not use Google's implementation. It looks pretty rubbish. The reason they use 10 bytes is they cannot be bothered to realise the last byte does not need a continuation bit. They also use a *terrible* implementation for deciding how long it needs to be. Here's a simple change which makes it 9 bytes, and easily optimised: the continuation bits are all shifted to the first byte, which effectively encodes the length in run-length encoding, i.e. the number of contiguous top order bits that are set to 1. i.e. {{length = Integer.numberOfLeadingZeros(firstByte ^ 0x)}} This way we read the first byte, and if there are any more to read, we read a long, and quickly truncate. was (Author: benedict): I'm confused as to why we need 10 bytes? Pretty much by definition a continuation bit encoding needs 9 bytes to represent 8 bytes. Let's not use Google's implementation. It looks pretty rubbish. The reason they use 10 bytes is they cannot be bothered to realise the last byte does not need a continuation bit. They also use a *terrible* implementation for deciding how long it needs to be. Here's a simple change which makes it 9 bytes, and easily optimised: the continuation bits are all shifted to the first byte, which effectively encodes the length in run-length encoding, i.e. the number of contiguous top order bits that are set to 1. i.e. {{length = Integer.numberOfLeadingZeros(firstByte ^ (byte) -1)}} This way we read the first byte, and if there are any more to read, we read a long, and quickly truncate. > Introduce writeVInt method to DataOutputStreamPlus > -- > > Key: CASSANDRA-9499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9499 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.0 beta 1 > > > CASSANDRA-8099 really could do with a writeVInt method, for both fixing > CASSANDRA-9498 but also efficiently encoding timestamp/deletion deltas. It > should be possible to make an especially efficient implementation against > BufferedDataOutputStreamPlus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9591) Scrub (recover) sstables even when -Index.db is missing
[ https://issues.apache.org/jira/browse/CASSANDRA-9591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589479#comment-14589479 ] Benedict commented on CASSANDRA-9591: - If it's impossible to have those values wired up, we can pass a flag into {{updateLiveSet()}} asking it not to build an interval tree in the case we're offline (which is detected by {{cfstore}} being {{null}} in the {{Tracker}}). > Scrub (recover) sstables even when -Index.db is missing > --- > > Key: CASSANDRA-9591 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9591 > Project: Cassandra > Issue Type: Improvement >Reporter: mck >Assignee: mck > Labels: sstablescrub > Fix For: 2.0.x > > Attachments: 9591-2.0.txt > > > Today SSTableReader needs at minimum 3 files to load an sstable: > - -Data.db > - -CompressionInfo.db > - -Index.db > But during the scrub process the -Index.db file isn't actually necessary, > unless there's corruption in the -Data.db and we want to be able to skip over > corrupted rows. Given that there is still a fair chance that there's nothing > wrong with the -Data.db file and we're just missing the -Index.db file this > patch addresses that situation. > So the following patch makes it possible for the StandaloneScrubber > (sstablescrub) to recover sstables despite missing -Index.db files. > This can happen from a catastrophic incident where data directories have been > lost and/or corrupted, or wiped and the backup not healthy. I'm aware that > normally one depends on replicas or snapshots to avoid such situations, but > such catastrophic incidents do occur in the wild. > I have not tested this patch against normal c* operations and all the other > (more critical) ways SSTableReader is used. i'll happily do that and add the > needed units tests if people see merit in accepting the patch. > Otherwise the patch can live with the issue, in-case anyone else needs it. > There's also a cassandra distribution bundled with the patch > [here|https://github.com/michaelsembwever/cassandra/releases/download/2.0.15-recover-sstables-without-indexdb/apache-cassandra-2.0.15-recover-sstables-without-indexdb.tar.gz] > to make life a little easier for anyone finding themselves in such a bad > situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9499) Introduce writeVInt method to DataOutputStreamPlus
[ https://issues.apache.org/jira/browse/CASSANDRA-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589470#comment-14589470 ] Benedict commented on CASSANDRA-9499: - I'm confused as to why we need 10 bytes? Pretty much by definition a continuation bit encoding needs 9 bytes to represent 8 bytes. Let's not use Google's implementation. It looks pretty rubbish. The reason they use 10 bytes is they cannot be bothered to realise the last byte does not need a continuation bit. They also use a *terrible* implementation for deciding how long it needs to be. Here's a simple change which makes it 9 bytes, and easily optimised: the continuation bits are all shifted to the first byte, which effectively encodes the length in run-length encoding, i.e. the number of contiguous top order bits that are set to 1. i.e. {{length = Integer.numberOfLeadingZeros(firstByte ^ (byte) -1)}} This way we read the first byte, and if there are any more to read, we read a long, and quickly truncate. > Introduce writeVInt method to DataOutputStreamPlus > -- > > Key: CASSANDRA-9499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9499 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.0 beta 1 > > > CASSANDRA-8099 really could do with a writeVInt method, for both fixing > CASSANDRA-9498 but also efficiently encoding timestamp/deletion deltas. It > should be possible to make an especially efficient implementation against > BufferedDataOutputStreamPlus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9499) Introduce writeVInt method to DataOutputStreamPlus
[ https://issues.apache.org/jira/browse/CASSANDRA-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589450#comment-14589450 ] Sylvain Lebresne commented on CASSANDRA-9499: - bq. breaking compatibility with persisted caches? We don't persist cache contents, we only persist the keys that are cached and reload them on startup, so this is a non-issue. More generally, I think we should just drop {{EncodedDataOutput}}/{{EncodedDataInput}} and just make sure we don't forget to update the serialization code from 8099 to use the new methods introduced here (which shouldn't take very long). > Introduce writeVInt method to DataOutputStreamPlus > -- > > Key: CASSANDRA-9499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9499 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Ariel Weisberg >Priority: Minor > Fix For: 3.0 beta 1 > > > CASSANDRA-8099 really could do with a writeVInt method, for both fixing > CASSANDRA-9498 but also efficiently encoding timestamp/deletion deltas. It > should be possible to make an especially efficient implementation against > BufferedDataOutputStreamPlus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)