[jira] [Created] (CASSANDRA-13845) Add keyspace and table name in schema validation exception
Jay Zhuang created CASSANDRA-13845: -- Summary: Add keyspace and table name in schema validation exception Key: CASSANDRA-13845 URL: https://issues.apache.org/jira/browse/CASSANDRA-13845 Project: Cassandra Issue Type: Improvement Reporter: Jay Zhuang Assignee: Jay Zhuang Priority: Minor We're seeing the following exception from time to time, it would be better to include keyspace table name, so we know which table update is causing the issue: {noformat} ERROR [InternalResponseStage:391] 2017-09-06 00:29:05,361 MigrationTask.java:96 - Configuration exception merging remote schema org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found af1f4650-9279-11e7-9df0-399587d0a542; expected a094fe70-89e3-11e7-b4d5-eb8faf28be34) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:785) ~[apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) ~[apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) ~[apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1391) ~[apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1347) ~[apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1297) ~[apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) ~[apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) [apache-cassandra-3.0.14.jar:3.0.14] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) [apache-cassandra-3.0.14.jar:3.0.14] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.14.jar:3.0.14] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121] {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13845) Add keyspace and table name in schema validation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Zhuang updated CASSANDRA-13845: --- Labels: lhf (was: ) > Add keyspace and table name in schema validation exception > -- > > Key: CASSANDRA-13845 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13845 > Project: Cassandra > Issue Type: Improvement >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Labels: lhf > > We're seeing the following exception from time to time, it would be better to > include keyspace table name, so we know which table update is causing the > issue: > {noformat} > ERROR [InternalResponseStage:391] 2017-09-06 00:29:05,361 > MigrationTask.java:96 - Configuration exception merging remote schema > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found af1f4650-9279-11e7-9df0-399587d0a542; expected > a094fe70-89e3-11e7-b4d5-eb8faf28be34) > at > org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:785) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:747) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at org.apache.cassandra.config.Schema.updateTable(Schema.java:661) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1391) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1347) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1297) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:92) > ~[apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) > [apache-cassandra-3.0.14.jar:3.0.14] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) > [apache-cassandra-3.0.14.jar:3.0.14] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_121] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_121] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_121] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) > [apache-cassandra-3.0.14.jar:3.0.14] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121] > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13760) presize collections
[ https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154822#comment-16154822 ] Jeff Jirsa commented on CASSANDRA-13760: Just realized that those are both fixed by {{e1aa7d32c95ff3f06de97803a186ff432237ecab}}, which isn't in your branch, so I've rebased and force-pushed. The new dtest will be @ https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/283/ > presize collections > --- > > Key: CASSANDRA-13760 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13760 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Dave Brosius >Assignee: Dave Brosius >Priority: Trivial > Fix For: 4.x > > Attachments: 0001-Presize-collections.patch, 13760.txt, > test-output.png > > > presize collections where sane, to avoid reallocs, or excess garbage -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13760) presize collections
[ https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154822#comment-16154822 ] Jeff Jirsa edited comment on CASSANDRA-13760 at 9/6/17 5:18 AM: Just realized that those are both fixed by {{e1aa7d32c95ff3f06de97803a186ff432237ecab}}, which isn't in (y)our branch, so I've rebased and [force-pushed|https://github.com/jeffjirsa/cassandra/commits/cassandra-13760]. The new dtest will be @ https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/283/ was (Author: jjirsa): Just realized that those are both fixed by {{e1aa7d32c95ff3f06de97803a186ff432237ecab}}, which isn't in your branch, so I've rebased and force-pushed. The new dtest will be @ https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/283/ > presize collections > --- > > Key: CASSANDRA-13760 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13760 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Dave Brosius >Assignee: Dave Brosius >Priority: Trivial > Fix For: 4.x > > Attachments: 0001-Presize-collections.patch, 13760.txt, > test-output.png > > > presize collections where sane, to avoid reallocs, or excess garbage -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13760) presize collections
[ https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154818#comment-16154818 ] Jeff Jirsa commented on CASSANDRA-13760: Attached screenshot of dtest results - 2 failures aren't typical failures: !test-output.png|thumbnail! Checked console output, {{repair_tests.incremental_repair_test.TestIncRepair.sstable_marking_test}} looks like a netty error: {code} Unexpected error in node2 log, error: WARN [epollEventLoopGroup-2-8] 2017-08-31 09:12:42,153 Slf4JLogger.java:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] {code} Meanwhile {{native_transport_ssl_test.NativeTransportSSL.connect_to_ssl_test}} throws an actual ssl error: {code} WARN [epollEventLoopGroup-2-7] 2017-08-31 07:46:04,104 Slf4JLogger.java:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 04100100160001000b43514c5f56455253494f4e0005332e302e30 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1057) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:934) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:397) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144] {code} > presize collections > --- > > Key: CASSANDRA-13760 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13760 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Dave Brosius >Assignee: Dave Brosius >Priority: Trivial > Fix For: 4.x > > Attachments: 0001-Presize-collections.patch, 13760.txt, > test-output.png > > > presize collections where sane, to avoid reallocs, or excess garbage -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13760) presize collections
[ https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13760: --- Attachment: test-output.png > presize collections > --- > > Key: CASSANDRA-13760 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13760 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Dave Brosius >Assignee: Dave Brosius >Priority: Trivial > Fix For: 4.x > > Attachments: 0001-Presize-collections.patch, 13760.txt, > test-output.png > > > presize collections where sane, to avoid reallocs, or excess garbage -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[ https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154716#comment-16154716 ] Kurt Greaves commented on CASSANDRA-11500: -- Looks good. Thanks for all the hard work [~jasonstack] and [~pauloricardomg] > Obsolete MV entry may not be properly deleted > - > > Key: CASSANDRA-11500 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11500 > Project: Cassandra > Issue Type: Bug > Components: Materialized Views >Reporter: Sylvain Lebresne >Assignee: ZhaoYang > Fix For: 3.0.15, 3.11.1, 4.0 > > > When a Materialized View uses a non-PK base table column in its PK, if an > update changes that column value, we add the new view entry and remove the > old one. When doing that removal, the current code uses the same timestamp > than for the liveness info of the new entry, which is the max timestamp for > any columns participating to the view PK. This is not correct for the > deletion as the old view entry could have other columns with higher timestamp > which won't be deleted as can easily shown by the failing of the following > test: > {noformat} > CREATE TABLE t (k int PRIMARY KEY, a int, b int); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (k, a); > INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0; > UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1; > UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; > SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old > (invalid) and the new one > {noformat} > So the correct timestamp to use for the deletion is the biggest timestamp in > the old view entry (which we know since we read the pre-existing base row), > and that is what CASSANDRA-11475 does (the test above thus doesn't fail on > that branch). > Unfortunately, even then we can still have problems if further updates > requires us to overide the old entry. Consider the following case: > {noformat} > CREATE TABLE t (k int PRIMARY KEY, a int, b int); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (k, a); > INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0; > UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1; > UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the > entry for a=1 with timestamp 10 > UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert > an entry for a=1 but shouldn't be deleted by the prior deletion > UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this > game more than once > UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1; > ... > {noformat} > In a way, this is saying that the "shadowable" deletion mechanism is not > general enough: we need to be able to re-insert an entry when a prior one had > been deleted before, but we can't rely on timestamps being strictly bigger on > the re-insert. In that sense, this can be though as a similar problem than > CASSANDRA-10965, though the solution there of a single flag is not enough > since we can have to replace more than once. > I think the proper solution would be to ship enough information to always be > able to decide when a view deletion is shadowed. Which means that both > liveness info (for updates) and shadowable deletion would need to ship the > timestamp of any base table column that is part the view PK (so {{a}} in the > example below). It's doable (and not that hard really), but it does require > a change to the sstable and intra-node protocol, which makes this a bit > painful right now. > But I'll also note that as CASSANDRA-1096 shows, the timestamp is not even > enough since on equal timestamp the value can be the deciding factor. So in > theory we'd have to ship the value of those columns (in the case of a > deletion at least since we have it in the view PK for updates). That said, on > that last problem, my preference would be that we start prioritizing > CASSANDRA-6123 seriously so we don't have to care about conflicting timestamp > anymore, which would make this problem go away. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154343#comment-16154343 ] Dimitar Dimitrov edited comment on CASSANDRA-13692 at 9/5/17 9:50 PM: -- I ended up rebasing the 3.0, 3.11, and trunk versions of the change, and the results are as follows: * No more consistency_test.TestConsistency.test_13747 failures for the 3.0 and 3.11 changes. Small differences between actual and expected dtest failures, which seem to be flaking out. * The trunk changes are still hitting what seems to be an existing problem plaguing cassci.datastax.com trunk dtest jobs for the last 10 days or so (see https://cassci.datastax.com/job/trunk_dtest/). ** The problem is currently being investigated. Now it looks like the 2.2, 3.0, and 3.11 changes can be accepted as passing the testall and dtest criteria, only the trunk changes need to be verified, after the problem affecting trunk gets resolved. I'll post an update once this happens, but in the meantime, it's possible to mark this as ready for review. was (Author: dimitarndimitrov): I ended up rebasing the 3.0, 3.11, and trunk versions of the change, and the results are as follows: * No more consistency_test.TestConsistency.test_13747 failures for the 3.0 and 3.11 changes. Small differences between actual and expected dtest failures, which seem to be flaking out. * The trunk changes are still hitting what seems to be an existing problem plaguing cassci.datastax.com trunk dtest jobs for the last 10 days or so (see https://cassci.datastax.com/job/trunk_dtest/). ** The problem is currently being investigated. Now it looks like the 2.2, 3.0, and 3.11 changes can be accepted as passing the testall and dtest criteria, only the trunk changes need to be verified, after the problem affecting trunk gets resolved. I'll post an update once this happens, but in the meantime, I'd assume it's safe to mark this as "Awaiting Feedback". > CompactionAwareWriter_getWriteDirectory throws incompatible exceptions > -- > > Key: CASSANDRA-13692 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13692 > Project: Cassandra > Issue Type: Bug > Components: Compaction >Reporter: Hao Zhong >Assignee: Dimitar Dimitrov > Labels: lhf > Attachments: c13692-2.2-dtest-results.PNG, > c13692-2.2-testall-results.PNG, c13692-3.0-dtest-results.PNG, > c13692-3.0-testall-results.PNG, c13692-3.11-dtest-results.PNG, > c13692-3.11-testall-results.PNG, c13692-dtest-results.PNG, > c13692-testall-results.PNG > > > The CompactionAwareWriter_getWriteDirectory throws RuntimeException: > {code} > public Directories.DataDirectory getWriteDirectory(Iterable > sstables, long estimatedWriteSize) > { > File directory = null; > for (SSTableReader sstable : sstables) > { > if (directory == null) > directory = sstable.descriptor.directory; > if (!directory.equals(sstable.descriptor.directory)) > { > logger.trace("All sstables not from the same disk - putting > results in {}", directory); > break; > } > } > Directories.DataDirectory d = > getDirectories().getDataDirectoryForFile(directory); > if (d != null) > { > long availableSpace = d.getAvailableSpace(); > if (availableSpace < estimatedWriteSize) > throw new RuntimeException(String.format("Not enough space to > write %s to %s (%s available)", > > FBUtilities.prettyPrintMemory(estimatedWriteSize), > d.location, > > FBUtilities.prettyPrintMemory(availableSpace))); > logger.trace("putting compaction results in {}", directory); > return d; > } > d = getDirectories().getWriteableLocation(estimatedWriteSize); > if (d == null) > throw new RuntimeException(String.format("Not enough disk space > to store %s", > > FBUtilities.prettyPrintMemory(estimatedWriteSize))); > return d; > } > {code} > However, the thrown exception does not trigger the failure policy. > CASSANDRA-11448 fixed a similar problem. The buggy code is: > {code} > protected Directories.DataDirectory getWriteDirectory(long writeSize) > { > Directories.DataDirectory directory = > getDirectories().getWriteableLocation(writeSize); > if (directory == null) > throw new RuntimeException("Insufficient disk space to write " + > writeSize + " byte
[jira] [Commented] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
[ https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154343#comment-16154343 ] Dimitar Dimitrov commented on CASSANDRA-13692: -- I ended up rebasing the 3.0, 3.11, and trunk versions of the change, and the results are as follows: * No more consistency_test.TestConsistency.test_13747 failures for the 3.0 and 3.11 changes. Small differences between actual and expected dtest failures, which seem to be flaking out. * The trunk changes are still hitting what seems to be an existing problem plaguing cassci.datastax.com trunk dtest jobs for the last 10 days or so (see https://cassci.datastax.com/job/trunk_dtest/). ** The problem is currently being investigated. Now it looks like the 2.2, 3.0, and 3.11 changes can be accepted as passing the testall and dtest criteria, only the trunk changes need to be verified, after the problem affecting trunk gets resolved. I'll post an update once this happens, but in the meantime, I'd assume it's safe to mark this as "Awaiting Feedback". > CompactionAwareWriter_getWriteDirectory throws incompatible exceptions > -- > > Key: CASSANDRA-13692 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13692 > Project: Cassandra > Issue Type: Bug > Components: Compaction >Reporter: Hao Zhong >Assignee: Dimitar Dimitrov > Labels: lhf > Attachments: c13692-2.2-dtest-results.PNG, > c13692-2.2-testall-results.PNG, c13692-3.0-dtest-results.PNG, > c13692-3.0-testall-results.PNG, c13692-3.11-dtest-results.PNG, > c13692-3.11-testall-results.PNG, c13692-dtest-results.PNG, > c13692-testall-results.PNG > > > The CompactionAwareWriter_getWriteDirectory throws RuntimeException: > {code} > public Directories.DataDirectory getWriteDirectory(Iterable > sstables, long estimatedWriteSize) > { > File directory = null; > for (SSTableReader sstable : sstables) > { > if (directory == null) > directory = sstable.descriptor.directory; > if (!directory.equals(sstable.descriptor.directory)) > { > logger.trace("All sstables not from the same disk - putting > results in {}", directory); > break; > } > } > Directories.DataDirectory d = > getDirectories().getDataDirectoryForFile(directory); > if (d != null) > { > long availableSpace = d.getAvailableSpace(); > if (availableSpace < estimatedWriteSize) > throw new RuntimeException(String.format("Not enough space to > write %s to %s (%s available)", > > FBUtilities.prettyPrintMemory(estimatedWriteSize), > d.location, > > FBUtilities.prettyPrintMemory(availableSpace))); > logger.trace("putting compaction results in {}", directory); > return d; > } > d = getDirectories().getWriteableLocation(estimatedWriteSize); > if (d == null) > throw new RuntimeException(String.format("Not enough disk space > to store %s", > > FBUtilities.prettyPrintMemory(estimatedWriteSize))); > return d; > } > {code} > However, the thrown exception does not trigger the failure policy. > CASSANDRA-11448 fixed a similar problem. The buggy code is: > {code} > protected Directories.DataDirectory getWriteDirectory(long writeSize) > { > Directories.DataDirectory directory = > getDirectories().getWriteableLocation(writeSize); > if (directory == null) > throw new RuntimeException("Insufficient disk space to write " + > writeSize + " bytes"); > return directory; > } > {code} > The fixed code is: > {code} > protected Directories.DataDirectory getWriteDirectory(long writeSize) > { > Directories.DataDirectory directory = > getDirectories().getWriteableLocation(writeSize); > if (directory == null) > throw new FSWriteError(new IOException("Insufficient disk space > to write " + writeSize + " bytes"), ""); > return directory; > } > {code} > The fixed code throws FSWE and triggers the failure policy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mck updated CASSANDRA-13418: Comment: was deleted (was: thanks for the patch. is going to make a big difference to a number of users that have append-only datamodels! -- Mick Semb Wever Australia The Last Pickle Apache Cassandra Consulting http://www.thelastpickle.com ) > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary >Assignee: Romain GERARD > Labels: twcs > Fix For: 3.11.1, 4.0 > > Attachments: twcs-cleanup.png > > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154249#comment-16154249 ] Mick Semb Wever commented on CASSANDRA-13418: - thanks for the patch. is going to make a big difference to a number of users that have append-only datamodels! -- Mick Semb Wever Australia The Last Pickle Apache Cassandra Consulting http://www.thelastpickle.com > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary >Assignee: Romain GERARD > Labels: twcs > Fix For: 3.11.1, 4.0 > > Attachments: twcs-cleanup.png > > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[ https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154191#comment-16154191 ] Jason Brown commented on CASSANDRA-13844: - Sure, but we're only taking critical bugs for 2.1/2.2 right now, and I'm trying to judge if this meets that criteria. > sstableloader doesn't support non default storage_port and ssl_storage_port > --- > > Key: CASSANDRA-13844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 2.2.11, 2.1.19 > > > Currently *storage_port* and *ssl_storage_port* are hardcoded to the > defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so > this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[ https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154185#comment-16154185 ] Eduard Tudenhoefner commented on CASSANDRA-13844: - [~jasobrown] honestly I haven't tried using the C* {{BulkLoader}} to connect to a 2.1/2.2 cluster. I just saw that it wasn't fixed in 2.1/2.2 and it seemed easy enough to do it > sstableloader doesn't support non default storage_port and ssl_storage_port > --- > > Key: CASSANDRA-13844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 2.2.11, 2.1.19 > > > Currently *storage_port* and *ssl_storage_port* are hardcoded to the > defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so > this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[ https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154179#comment-16154179 ] Jason Brown commented on CASSANDRA-13844: - [~eduard.tudenhoefner] you can't use the 3.0 {{BulkLoader}} to connect to a 2.1/2.2 cluster, correct? > sstableloader doesn't support non default storage_port and ssl_storage_port > --- > > Key: CASSANDRA-13844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 2.2.11, 2.1.19 > > > Currently *storage_port* and *ssl_storage_port* are hardcoded to the > defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so > this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13618) CassandraRoleManager setup task improvement
[ https://issues.apache.org/jira/browse/CASSANDRA-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154147#comment-16154147 ] Jeff Jirsa commented on CASSANDRA-13618: Didnt get a chance to triage dtests before they rolled over back in July, so kicked off anew: 3.0 - https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/280/ 3.11 - https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/281/ trunk - https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/282/ > CassandraRoleManager setup task improvement > --- > > Key: CASSANDRA-13618 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13618 > Project: Cassandra > Issue Type: Bug > Components: Auth >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa > Fix For: 3.0.x, 3.11.x, 4.x > > > {{CassandraRoleManager}} blocks some functionality during setup, using a > delay added in CASSANDRA-9761 . Unfortunately, this setup is scheduled for > 10s after startup, and may not be necessary, meaning immediately after > startup some auth related queries may not behave as intended. We can skip > this delay without any additional risk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13603) Change repair midpoint logging from CASSANDRA-13052
[ https://issues.apache.org/jira/browse/CASSANDRA-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13603: --- Resolution: Fixed Fix Version/s: (was: 3.11.x) (was: 4.x) (was: 3.0.x) 4.0 3.11.1 3.0.15 Status: Resolved (was: Ready to Commit) Thanks. Committed as {{bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283}} to 3.0 and merged 3.11 and trunk. > Change repair midpoint logging from CASSANDRA-13052 > > > Key: CASSANDRA-13603 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13603 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Trivial > Fix For: 3.0.15, 3.11.1, 4.0 > > > In CASSANDRA-13052 , we changed the way we handle repairs on small ranges to > make them more sane in general, but {{MerkleTree.differenceHelper}} now > erroneously logs at error. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/90b39dc6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/90b39dc6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/90b39dc6 Branch: refs/heads/trunk Commit: 90b39dc619d75096e897fb8e90dd38b04e67f848 Parents: 3f5f702 bc5c231 Author: Jeff Jirsa Authored: Tue Sep 5 11:58:31 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:59:25 2017 -0700 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/MerkleTree.java | 23 + .../apache/cassandra/utils/MerkleTreeTest.java | 34 3 files changed, 51 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/CHANGES.txt -- diff --cc CHANGES.txt index 5a53e50,4302fdf..75d9bd6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,14 -1,5 +1,15 @@@ -3.0.15 +3.11.1 + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: + * Change repair midpoint logging for tiny ranges (CASSANDRA-13603) * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/src/java/org/apache/cassandra/utils/MerkleTree.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dcef7c7e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dcef7c7e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dcef7c7e Branch: refs/heads/trunk Commit: dcef7c7ed9bac3484322ed5b20eaa20d373833e7 Parents: fa36044 90b39dc Author: Jeff Jirsa Authored: Tue Sep 5 11:59:35 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 12:00:25 2017 -0700 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/MerkleTree.java | 23 + .../apache/cassandra/utils/MerkleTreeTest.java | 34 3 files changed, 51 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/dcef7c7e/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/dcef7c7e/src/java/org/apache/cassandra/utils/MerkleTree.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[2/6] cassandra git commit: Change repair midpoint logging for tiny ranges
Change repair midpoint logging for tiny ranges Patch by Jeff Jirsa; Reviewed by Blake Eggleston for CASSANDRA-13603 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bc5c2316 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bc5c2316 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bc5c2316 Branch: refs/heads/cassandra-3.11 Commit: bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283 Parents: 0493545 Author: Jeff Jirsa Authored: Tue Sep 5 11:57:58 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:57:58 2017 -0700 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/MerkleTree.java | 23 + .../apache/cassandra/utils/MerkleTreeTest.java | 34 3 files changed, 51 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a3eccf2..4302fdf 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * Change repair midpoint logging for tiny ranges (CASSANDRA-13603) * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/src/java/org/apache/cassandra/utils/MerkleTree.java -- diff --git a/src/java/org/apache/cassandra/utils/MerkleTree.java b/src/java/org/apache/cassandra/utils/MerkleTree.java index 0d5a469..22b61e8 100644 --- a/src/java/org/apache/cassandra/utils/MerkleTree.java +++ b/src/java/org/apache/cassandra/utils/MerkleTree.java @@ -22,6 +22,7 @@ import java.io.IOException; import java.io.Serializable; import java.util.*; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import com.google.common.collect.PeekingIterator; @@ -246,12 +247,20 @@ public class MerkleTree implements Serializable if (lhash != null && rhash != null && !Arrays.equals(lhash, rhash)) { -logger.debug("Digest mismatch detected, traversing trees [{}, {}]", ltree, rtree); -if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, active)) +if(lnode instanceof Leaf || rnode instanceof Leaf) { -logger.debug("Range {} fully inconsistent", active); +logger.debug("Digest mismatch detected among leaf nodes {}, {}", lnode, rnode); diff.add(active); } +else +{ +logger.debug("Digest mismatch detected, traversing trees [{}, {}]", ltree, rtree); +if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, active)) +{ +logger.debug("Range {} fully inconsistent", active); +diff.add(active); +} +} } else if (lhash == null || rhash == null) diff.add(active); @@ -265,6 +274,7 @@ public class MerkleTree implements Serializable * Takes two trees and a range for which they have hashes, but are inconsistent. * @return FULLY_INCONSISTENT if active is inconsistent, PARTIALLY_INCONSISTENT if only a subrange is inconsistent. */ +@VisibleForTesting static int differenceHelper(MerkleTree ltree, MerkleTree rtree, List diff, TreeRange active) { if (active.depth == Byte.MAX_VALUE) @@ -274,10 +284,9 @@ public class MerkleTree implements Serializable // sanity check for midpoint calculation, see CASSANDRA-13052 if (midpoint.equals(active.left) || midpoint.equals(active.right)) { -// Unfortunately we can't throw here to abort the validation process, as the code is executed in it's own -// thread with the caller waiting for a condition to be signaled after completion and without an option -// to indicate an error (2.x only). -logger.error("Invalid midpoint {} for [{},{}], range will be reported inconsistent", midpoint, active.left, active.right); +// If the midpoint equals either the left or the right, we have a range that's too small to split - we'll simply report the +// whole range as inconsistent +logger.debug("({}) No sane midpoint ({}) for range {} , marking whole range as inconsistent", active.depth, midpoint, active); return FULLY_INCONSISTENT; } http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/test/unit/org/apache/cassandra/utils/MerkleTreeTest.java
[1/6] cassandra git commit: Change repair midpoint logging for tiny ranges
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 0493545dd -> bc5c2316c refs/heads/cassandra-3.11 3f5f70281 -> 90b39dc61 refs/heads/trunk fa36044a5 -> dcef7c7ed Change repair midpoint logging for tiny ranges Patch by Jeff Jirsa; Reviewed by Blake Eggleston for CASSANDRA-13603 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bc5c2316 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bc5c2316 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bc5c2316 Branch: refs/heads/cassandra-3.0 Commit: bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283 Parents: 0493545 Author: Jeff Jirsa Authored: Tue Sep 5 11:57:58 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:57:58 2017 -0700 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/MerkleTree.java | 23 + .../apache/cassandra/utils/MerkleTreeTest.java | 34 3 files changed, 51 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a3eccf2..4302fdf 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * Change repair midpoint logging for tiny ranges (CASSANDRA-13603) * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/src/java/org/apache/cassandra/utils/MerkleTree.java -- diff --git a/src/java/org/apache/cassandra/utils/MerkleTree.java b/src/java/org/apache/cassandra/utils/MerkleTree.java index 0d5a469..22b61e8 100644 --- a/src/java/org/apache/cassandra/utils/MerkleTree.java +++ b/src/java/org/apache/cassandra/utils/MerkleTree.java @@ -22,6 +22,7 @@ import java.io.IOException; import java.io.Serializable; import java.util.*; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import com.google.common.collect.PeekingIterator; @@ -246,12 +247,20 @@ public class MerkleTree implements Serializable if (lhash != null && rhash != null && !Arrays.equals(lhash, rhash)) { -logger.debug("Digest mismatch detected, traversing trees [{}, {}]", ltree, rtree); -if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, active)) +if(lnode instanceof Leaf || rnode instanceof Leaf) { -logger.debug("Range {} fully inconsistent", active); +logger.debug("Digest mismatch detected among leaf nodes {}, {}", lnode, rnode); diff.add(active); } +else +{ +logger.debug("Digest mismatch detected, traversing trees [{}, {}]", ltree, rtree); +if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, active)) +{ +logger.debug("Range {} fully inconsistent", active); +diff.add(active); +} +} } else if (lhash == null || rhash == null) diff.add(active); @@ -265,6 +274,7 @@ public class MerkleTree implements Serializable * Takes two trees and a range for which they have hashes, but are inconsistent. * @return FULLY_INCONSISTENT if active is inconsistent, PARTIALLY_INCONSISTENT if only a subrange is inconsistent. */ +@VisibleForTesting static int differenceHelper(MerkleTree ltree, MerkleTree rtree, List diff, TreeRange active) { if (active.depth == Byte.MAX_VALUE) @@ -274,10 +284,9 @@ public class MerkleTree implements Serializable // sanity check for midpoint calculation, see CASSANDRA-13052 if (midpoint.equals(active.left) || midpoint.equals(active.right)) { -// Unfortunately we can't throw here to abort the validation process, as the code is executed in it's own -// thread with the caller waiting for a condition to be signaled after completion and without an option -// to indicate an error (2.x only). -logger.error("Invalid midpoint {} for [{},{}], range will be reported inconsistent", midpoint, active.left, active.right); +// If the midpoint equals either the left or the right, we have a range that's too small to split - we'll simply report the +// whole range as inconsistent +logger.debug("({}) No sane midpoint ({}) for range {} , marking whole range as inconsistent", active.depth, midpoint, active); retur
[3/6] cassandra git commit: Change repair midpoint logging for tiny ranges
Change repair midpoint logging for tiny ranges Patch by Jeff Jirsa; Reviewed by Blake Eggleston for CASSANDRA-13603 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bc5c2316 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bc5c2316 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bc5c2316 Branch: refs/heads/trunk Commit: bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283 Parents: 0493545 Author: Jeff Jirsa Authored: Tue Sep 5 11:57:58 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:57:58 2017 -0700 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/MerkleTree.java | 23 + .../apache/cassandra/utils/MerkleTreeTest.java | 34 3 files changed, 51 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a3eccf2..4302fdf 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * Change repair midpoint logging for tiny ranges (CASSANDRA-13603) * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/src/java/org/apache/cassandra/utils/MerkleTree.java -- diff --git a/src/java/org/apache/cassandra/utils/MerkleTree.java b/src/java/org/apache/cassandra/utils/MerkleTree.java index 0d5a469..22b61e8 100644 --- a/src/java/org/apache/cassandra/utils/MerkleTree.java +++ b/src/java/org/apache/cassandra/utils/MerkleTree.java @@ -22,6 +22,7 @@ import java.io.IOException; import java.io.Serializable; import java.util.*; +import com.google.common.annotations.VisibleForTesting; import com.google.common.base.Preconditions; import com.google.common.collect.PeekingIterator; @@ -246,12 +247,20 @@ public class MerkleTree implements Serializable if (lhash != null && rhash != null && !Arrays.equals(lhash, rhash)) { -logger.debug("Digest mismatch detected, traversing trees [{}, {}]", ltree, rtree); -if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, active)) +if(lnode instanceof Leaf || rnode instanceof Leaf) { -logger.debug("Range {} fully inconsistent", active); +logger.debug("Digest mismatch detected among leaf nodes {}, {}", lnode, rnode); diff.add(active); } +else +{ +logger.debug("Digest mismatch detected, traversing trees [{}, {}]", ltree, rtree); +if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, active)) +{ +logger.debug("Range {} fully inconsistent", active); +diff.add(active); +} +} } else if (lhash == null || rhash == null) diff.add(active); @@ -265,6 +274,7 @@ public class MerkleTree implements Serializable * Takes two trees and a range for which they have hashes, but are inconsistent. * @return FULLY_INCONSISTENT if active is inconsistent, PARTIALLY_INCONSISTENT if only a subrange is inconsistent. */ +@VisibleForTesting static int differenceHelper(MerkleTree ltree, MerkleTree rtree, List diff, TreeRange active) { if (active.depth == Byte.MAX_VALUE) @@ -274,10 +284,9 @@ public class MerkleTree implements Serializable // sanity check for midpoint calculation, see CASSANDRA-13052 if (midpoint.equals(active.left) || midpoint.equals(active.right)) { -// Unfortunately we can't throw here to abort the validation process, as the code is executed in it's own -// thread with the caller waiting for a condition to be signaled after completion and without an option -// to indicate an error (2.x only). -logger.error("Invalid midpoint {} for [{},{}], range will be reported inconsistent", midpoint, active.left, active.right); +// If the midpoint equals either the left or the right, we have a range that's too small to split - we'll simply report the +// whole range as inconsistent +logger.debug("({}) No sane midpoint ({}) for range {} , marking whole range as inconsistent", active.depth, midpoint, active); return FULLY_INCONSISTENT; } http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/test/unit/org/apache/cassandra/utils/MerkleTreeTest.java -
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/90b39dc6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/90b39dc6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/90b39dc6 Branch: refs/heads/cassandra-3.11 Commit: 90b39dc619d75096e897fb8e90dd38b04e67f848 Parents: 3f5f702 bc5c231 Author: Jeff Jirsa Authored: Tue Sep 5 11:58:31 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:59:25 2017 -0700 -- CHANGES.txt | 1 + .../org/apache/cassandra/utils/MerkleTree.java | 23 + .../apache/cassandra/utils/MerkleTreeTest.java | 34 3 files changed, 51 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/CHANGES.txt -- diff --cc CHANGES.txt index 5a53e50,4302fdf..75d9bd6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,14 -1,5 +1,15 @@@ -3.0.15 +3.11.1 + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: + * Change repair midpoint logging for tiny ranges (CASSANDRA-13603) * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/src/java/org/apache/cassandra/utils/MerkleTree.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11995) Commitlog replaced with all NULs
[ https://issues.apache.org/jira/browse/CASSANDRA-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-11995: --- Resolution: Fixed Reviewer: Branimir Lambov Fix Version/s: 4.0 3.11.1 3.0.15 Status: Resolved (was: Ready to Commit) Thanks for the 'ready-to-commit' change. Committed as {{0493545dd08d29c34d757bb2f1d90052d03d24c6}} to 3.0 and merged up through 3.11 and trunk. > Commitlog replaced with all NULs > > > Key: CASSANDRA-11995 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11995 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Windows 10 Enterprise 1511 > DataStax Cassandra Community Server 2.2.3 >Reporter: James Howe >Assignee: Jeff Jirsa > Fix For: 3.0.15, 3.11.1, 4.0 > > > I noticed this morning that Cassandra was failing to start, after being shut > down on Friday. > {code} > ERROR 09:13:37 Exiting due to error while processing commit log during > initialization. > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file C:\Program Files\DataStax > Community\data\commitlog\CommitLog-5-1465571056722.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:302) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:273) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:513) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) > [apache-cassandra-2.2.3.jar:2.2.3] > {code} > Checking the referenced file reveals it comprises 33,554,432 (32 * 1024 * > 1024) NUL bytes. > No logs (stdout, stderr, prunsrv) from the shutdown show any other issues and > appear exactly as normal. > Is installed as a service via DataStax's distribution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fa36044a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fa36044a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fa36044a Branch: refs/heads/trunk Commit: fa36044a585d2268d6dd61603f3fb21e6b8ec286 Parents: 733ea88 3f5f702 Author: Jeff Jirsa Authored: Tue Sep 5 11:48:04 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:48:40 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/commitlog/CommitLogReader.java | 3 ++- .../apache/cassandra/db/commitlog/CommitLogTest.java| 12 +++- 3 files changed, 14 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fa36044a/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fa36044a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java -- diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java index 301c832,8c04329..864325b --- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java @@@ -147,6 -149,7 +147,7 @@@ public class CommitLogReade if (desc == null) { // don't care about whether or not the handler thinks we can continue. We can't w/out descriptor. -// whether or not we continue with startup will depend on whether this is the last segment ++// whether or not we can continue depends on whether this is the last segment handler.handleUnrecoverableError(new CommitLogReadException( String.format("Could not read commit log descriptor in file %s", file), CommitLogReadErrorReason.UNRECOVERABLE_DESCRIPTOR_ERROR, http://git-wip-us.apache.org/repos/asf/cassandra/blob/fa36044a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java -- diff --cc test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java index f98dd6b,aab55a5..956a05d --- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java +++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java @@@ -153,9 -155,22 +156,16 @@@ public class CommitLogTes } @Test -public void testRecoveryWithFinalEmptyLog() throws Exception ++public void testRecoveryWithEmptyFinalLog() throws Exception + { -// Even though it's empty, it's the last commitlog segment, so allowTruncation=true should allow it to pass -CommitLog.instance.recoverFiles(new File[]{tmpFile(CommitLogDescriptor.current_version)}); ++ CommitLog.instance.recoverFiles(tmpFile(CommitLogDescriptor.current_version)); + } + -@Test -public void testRecoveryWithEmptyLog20() throws Exception -{ - CommitLog.instance.recoverFiles(tmpFile(CommitLogDescriptor.VERSION_20)); -} + + @Test public void testRecoveryWithZeroLog() throws Exception { -testRecovery(new byte[10], null); +testRecovery(new byte[10], CommitLogReplayException.class); } @Test - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3f5f7028 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3f5f7028 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3f5f7028 Branch: refs/heads/trunk Commit: 3f5f70281ee87c09d312d2169d6335e5b0abf815 Parents: d44a0d2 0493545 Author: Jeff Jirsa Authored: Tue Sep 5 11:47:18 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:47:55 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/commitlog/CommitLogReader.java | 3 ++- .../apache/cassandra/db/commitlog/CommitLogTest.java| 12 +++- 3 files changed, 14 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/CHANGES.txt -- diff --cc CHANGES.txt index 0faed3d,a3eccf2..5a53e50 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,18 -1,10 +1,19 @@@ -3.0.15 +3.11.1 + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: + * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) * Fix race condition in read command serialization (CASSANDRA-13363) - * Enable segement creation before recovering commitlogs (CASSANDRA-13587) * Fix AssertionError in short read protection (CASSANDRA-13747) * Don't skip corrupted sstables on startup (CASSANDRA-13620) * Fix the merging of cells with different user type versions (CASSANDRA-13776) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java -- diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java index d1cb8d6,000..8c04329 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java @@@ -1,514 -1,0 +1,515 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.db.commitlog; + +import java.io.*; +import java.util.*; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.zip.CRC32; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.cassandra.config.DatabaseDescriptor; +import org.apache.cassandra.db.Mutation; +import org.apache.cassandra.db.UnknownColumnFamilyException; +import org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadErrorReason; +import org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadException; +import org.apache.cassandra.db.partitions.PartitionUpdate; +import org.apache.cassandra.db.rows.SerializationHelper; +import org.apache.cassandra.io.util.DataInputBuffer; +import org.apache.cassandra.io.util.FileDataInput; +import org.apache.cassandra.io.util.RandomAccessRead
[2/6] cassandra git commit: On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of t
On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of the final segment Patch by Jeff Jirsa; Reviewed by Branimir Lambov for CASSANDRA-11995 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0493545d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0493545d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0493545d Branch: refs/heads/cassandra-3.11 Commit: 0493545dd08d29c34d757bb2f1d90052d03d24c6 Parents: ef5ac1a Author: Jeff Jirsa Authored: Fri Mar 17 13:54:03 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:46:03 2017 -0700 -- CHANGES.txt| 1 + .../cassandra/db/commitlog/CommitLogReplayer.java | 4 +++- .../apache/cassandra/db/commitlog/CommitLogTest.java | 13 - 3 files changed, 16 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 853bf61..a3eccf2 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java -- diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java index 205c36a..b3b26dd 100644 --- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java @@ -355,7 +355,9 @@ public class CommitLogReplayer desc = null; } if (desc == null) { -handleReplayError(false, "Could not read commit log descriptor in file %s", file); +// Presumably a failed CRC or other IO error occurred, which may be ok if it's the last segment +// where we tolerate (and expect) truncation +handleReplayError(tolerateTruncation, "Could not read commit log descriptor in file %s", file); return; } if (segmentId != desc.id) http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java -- diff --git a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java index 90dc258..1543415 100644 --- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java +++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java @@ -117,13 +117,24 @@ public class CommitLogTest @Test public void testRecoveryWithEmptyLog() throws Exception { +// The first empty file we expect to throw as it's invalid +// We need to pass the second as well, because allowTruncation will be set to true for the final segment runExpecting(() -> { -CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.current_version) }); +CommitLog.instance.recover(new File[]{ +tmpFile(CommitLogDescriptor.current_version), +tmpFile(CommitLogDescriptor.current_version) }); return null; }, CommitLogReplayException.class); } @Test +public void testRecoveryWithEmptyFinalLog() throws Exception +{ +// Even though it's empty, it's the last commitlog segment, so allowTruncation=true should allow it to pass +CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.current_version) }); +} + +@Test public void testRecoveryWithEmptyLog20() throws Exception { CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.VERSION_20) }); - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[1/6] cassandra git commit: On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of t
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 ef5ac1a4a -> 0493545dd refs/heads/cassandra-3.11 d44a0d25b -> 3f5f70281 refs/heads/trunk 733ea88fa -> fa36044a5 On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of the final segment Patch by Jeff Jirsa; Reviewed by Branimir Lambov for CASSANDRA-11995 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0493545d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0493545d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0493545d Branch: refs/heads/cassandra-3.0 Commit: 0493545dd08d29c34d757bb2f1d90052d03d24c6 Parents: ef5ac1a Author: Jeff Jirsa Authored: Fri Mar 17 13:54:03 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:46:03 2017 -0700 -- CHANGES.txt| 1 + .../cassandra/db/commitlog/CommitLogReplayer.java | 4 +++- .../apache/cassandra/db/commitlog/CommitLogTest.java | 13 - 3 files changed, 16 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 853bf61..a3eccf2 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java -- diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java index 205c36a..b3b26dd 100644 --- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java @@ -355,7 +355,9 @@ public class CommitLogReplayer desc = null; } if (desc == null) { -handleReplayError(false, "Could not read commit log descriptor in file %s", file); +// Presumably a failed CRC or other IO error occurred, which may be ok if it's the last segment +// where we tolerate (and expect) truncation +handleReplayError(tolerateTruncation, "Could not read commit log descriptor in file %s", file); return; } if (segmentId != desc.id) http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java -- diff --git a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java index 90dc258..1543415 100644 --- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java +++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java @@ -117,13 +117,24 @@ public class CommitLogTest @Test public void testRecoveryWithEmptyLog() throws Exception { +// The first empty file we expect to throw as it's invalid +// We need to pass the second as well, because allowTruncation will be set to true for the final segment runExpecting(() -> { -CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.current_version) }); +CommitLog.instance.recover(new File[]{ +tmpFile(CommitLogDescriptor.current_version), +tmpFile(CommitLogDescriptor.current_version) }); return null; }, CommitLogReplayException.class); } @Test +public void testRecoveryWithEmptyFinalLog() throws Exception +{ +// Even though it's empty, it's the last commitlog segment, so allowTruncation=true should allow it to pass +CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.current_version) }); +} + +@Test public void testRecoveryWithEmptyLog20() throws Exception { CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.VERSION_20) }); - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3f5f7028 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3f5f7028 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3f5f7028 Branch: refs/heads/cassandra-3.11 Commit: 3f5f70281ee87c09d312d2169d6335e5b0abf815 Parents: d44a0d2 0493545 Author: Jeff Jirsa Authored: Tue Sep 5 11:47:18 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:47:55 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/commitlog/CommitLogReader.java | 3 ++- .../apache/cassandra/db/commitlog/CommitLogTest.java| 12 +++- 3 files changed, 14 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/CHANGES.txt -- diff --cc CHANGES.txt index 0faed3d,a3eccf2..5a53e50 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,18 -1,10 +1,19 @@@ -3.0.15 +3.11.1 + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: + * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) * Fix race condition in read command serialization (CASSANDRA-13363) - * Enable segement creation before recovering commitlogs (CASSANDRA-13587) * Fix AssertionError in short read protection (CASSANDRA-13747) * Don't skip corrupted sstables on startup (CASSANDRA-13620) * Fix the merging of cells with different user type versions (CASSANDRA-13776) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java -- diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java index d1cb8d6,000..8c04329 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java @@@ -1,514 -1,0 +1,515 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.db.commitlog; + +import java.io.*; +import java.util.*; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.zip.CRC32; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.cassandra.config.DatabaseDescriptor; +import org.apache.cassandra.db.Mutation; +import org.apache.cassandra.db.UnknownColumnFamilyException; +import org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadErrorReason; +import org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadException; +import org.apache.cassandra.db.partitions.PartitionUpdate; +import org.apache.cassandra.db.rows.SerializationHelper; +import org.apache.cassandra.io.util.DataInputBuffer; +import org.apache.cassandra.io.util.FileDataInput; +import org.apache.cassandra.io.util.RandomA
[3/6] cassandra git commit: On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of t
On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of the final segment Patch by Jeff Jirsa; Reviewed by Branimir Lambov for CASSANDRA-11995 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0493545d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0493545d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0493545d Branch: refs/heads/trunk Commit: 0493545dd08d29c34d757bb2f1d90052d03d24c6 Parents: ef5ac1a Author: Jeff Jirsa Authored: Fri Mar 17 13:54:03 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 11:46:03 2017 -0700 -- CHANGES.txt| 1 + .../cassandra/db/commitlog/CommitLogReplayer.java | 4 +++- .../apache/cassandra/db/commitlog/CommitLogTest.java | 13 - 3 files changed, 16 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 853bf61..a3eccf2 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * Better handle corrupt final commitlog segment (CASSANDRA-11995) * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java -- diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java index 205c36a..b3b26dd 100644 --- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java @@ -355,7 +355,9 @@ public class CommitLogReplayer desc = null; } if (desc == null) { -handleReplayError(false, "Could not read commit log descriptor in file %s", file); +// Presumably a failed CRC or other IO error occurred, which may be ok if it's the last segment +// where we tolerate (and expect) truncation +handleReplayError(tolerateTruncation, "Could not read commit log descriptor in file %s", file); return; } if (segmentId != desc.id) http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java -- diff --git a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java index 90dc258..1543415 100644 --- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java +++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java @@ -117,13 +117,24 @@ public class CommitLogTest @Test public void testRecoveryWithEmptyLog() throws Exception { +// The first empty file we expect to throw as it's invalid +// We need to pass the second as well, because allowTruncation will be set to true for the final segment runExpecting(() -> { -CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.current_version) }); +CommitLog.instance.recover(new File[]{ +tmpFile(CommitLogDescriptor.current_version), +tmpFile(CommitLogDescriptor.current_version) }); return null; }, CommitLogReplayException.class); } @Test +public void testRecoveryWithEmptyFinalLog() throws Exception +{ +// Even though it's empty, it's the last commitlog segment, so allowTruncation=true should allow it to pass +CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.current_version) }); +} + +@Test public void testRecoveryWithEmptyLog20() throws Exception { CommitLog.instance.recover(new File[]{ tmpFile(CommitLogDescriptor.VERSION_20) }); - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13756) StreamingHistogram is not thread safe
[ https://issues.apache.org/jira/browse/CASSANDRA-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13756: --- Resolution: Fixed Fix Version/s: (was: 3.11.x) (was: 3.0.x) 3.11.1 3.0.15 Status: Resolved (was: Ready to Commit) Thanks all. Marcus, I've addressed your comments on commit. Committed as {{ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2}} to 3.0 and 3.11, and {{merge -s ours}} to trunk (no changes but CHANGES.txt) with Jason and Marcus both listed as reviewers. > StreamingHistogram is not thread safe > - > > Key: CASSANDRA-13756 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13756 > Project: Cassandra > Issue Type: Bug >Reporter: xiangzhou xia >Assignee: Jeff Jirsa > Fix For: 3.0.15, 3.11.1 > > > When we test C*3 in shadow cluster, we notice after a period of time, several > data node suddenly run into 100% cpu and stop process query anymore. > After investigation, we found that threads are stuck on the sum() in > streaminghistogram class. Those are jmx threads that working on expose > getTombStoneRatio metrics (since jmx is kicked off every 3 seconds, there is > a chance that multiple jmx thread is access streaminghistogram at the same > time). > After further investigation, we find that the optimization in CASSANDRA-13038 > led to a spool flush every time when we call sum(). Since TreeMap is not > thread safe, threads will be stuck when multiple threads visit sum() at the > same time. > There are two approaches to solve this issue. > The first one is to add a lock to the flush in sum() which will introduce > some extra overhead to streaminghistogram. > The second one is to avoid streaminghistogram to be access by multiple > threads. For our specific case, is to remove the metrics we added. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d44a0d25 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d44a0d25 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d44a0d25 Branch: refs/heads/trunk Commit: d44a0d25bb9a07f5e43a9951bd13f8259dfccd36 Parents: 03155b0 ef5ac1a Author: Jeff Jirsa Authored: Tue Sep 5 09:54:36 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 09:55:26 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 2 +- .../io/sstable/metadata/MetadataCollector.java | 10 +- .../cassandra/utils/StreamingHistogram.java | 333 ++- .../microbench/StreamingHistogramBench.java | 192 ++- .../cassandra/utils/StreamingHistogramTest.java | 43 +-- 6 files changed, 306 insertions(+), 275 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/CHANGES.txt -- diff --cc CHANGES.txt index 7056075,853bf61..0faed3d --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,17 -1,9 +1,18 @@@ -3.0.15 +3.11.1 + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: + * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) * Fix race condition in read command serialization (CASSANDRA-13363) - * Enable segement creation before recovering commitlogs (CASSANDRA-13587) * Fix AssertionError in short read protection (CASSANDRA-13747) * Don't skip corrupted sstables on startup (CASSANDRA-13620) * Fix the merging of cells with different user type versions (CASSANDRA-13776) http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/utils/StreamingHistogram.java -- diff --cc src/java/org/apache/cassandra/utils/StreamingHistogram.java index 9114c7d,6500a1a..df49d8d --- a/src/java/org/apache/cassandra/utils/StreamingHistogram.java +++ b/src/java/org/apache/cassandra/utils/StreamingHistogram.java @@@ -24,6 -24,6 +24,7 @@@ import com.google.common.base.Objects import org.apache.cassandra.db.TypeSizes; import org.apache.cassandra.io.ISerializer; ++import org.apache.cassandra.io.sstable.SSTable; import org.apache.cassandra.io.util.DataInputPlus; import org.apache.cassandra.io.util.DataOutputPlus; @@@ -39,172 -39,15 +40,33 @@@ public class StreamingHistogra public static final StreamingHistogramSerializer serializer = new StreamingHistogramSerializer(); // TreeMap to hold bins of histogram. -private final TreeMap bin; +// The key is a numeric type so we can avoid boxing/unboxing streams of different key types +// The value is a unboxed long array always of length == 1 +// Serialized Histograms always writes with double keys for backwards compatibility +private final TreeMap bin; + - // Keep a second, larger buffer to spool data in, before finalizing it into `bin` - private final TreeMap spool; - +// maximum bin size for this histogram private final int maxBinSize; - // maximum size of the spool - private final int maxSpoolSize; - - // voluntarily give up resolution for speed - private final int roundSeconds; -private StreamingHistogram(int maxBinSize, Map bin) + +/** + * Creates a new histogram with max bin size of maxBinSize + * @par
[2/6] cassandra git commit: StreamingHistogram is not thread safe
StreamingHistogram is not thread safe Patch by Jeff Jirsa; Reviewed by Jason Brown, Marcus Eriksson for CASSANDRA-13756 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ef5ac1a4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ef5ac1a4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ef5ac1a4 Branch: refs/heads/cassandra-3.11 Commit: ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2 Parents: 1e80e35 Author: Jeff Jirsa Authored: Tue Aug 29 10:31:16 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 09:54:27 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 2 +- .../io/sstable/metadata/MetadataCollector.java | 10 +- .../cassandra/utils/StreamingHistogram.java | 284 ++- .../microbench/StreamingHistogramBench.java | 192 +++-- .../cassandra/utils/StreamingHistogramTest.java | 31 +- 6 files changed, 263 insertions(+), 257 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 1ed3fe6..853bf61 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) * Fix race condition in read command serialization (CASSANDRA-13363) http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 183176c..983d6b1 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -2415,7 +2415,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean long allColumns = 0; int localTime = (int)(System.currentTimeMillis()/1000); -for (SSTableReader sstable : getSSTables(SSTableSet.LIVE)) +for (SSTableReader sstable : getSSTables(SSTableSet.CANONICAL)) { allDroppable += sstable.getDroppableTombstonesBefore(localTime - sstable.metadata.params.gcGraceSeconds); allColumns += sstable.getEstimatedColumnCount().mean() * sstable.getEstimatedColumnCount().count(); http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java index 3b13cf4..487a932 100644 --- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java +++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java @@ -59,9 +59,9 @@ public class MetadataCollector implements PartitionStatisticsCollector return new EstimatedHistogram(150); } -static StreamingHistogram defaultTombstoneDropTimeHistogram() +static StreamingHistogram.StreamingHistogramBuilder defaultTombstoneDropTimeHistogramBuilder() { -return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS); +return new StreamingHistogram.StreamingHistogramBuilder(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS); } public static StatsMetadata defaultStatsMetadata() @@ -76,7 +76,7 @@ public class MetadataCollector implements PartitionStatisticsCollector 0, Integer.MAX_VALUE, NO_COMPRESSION_RATIO, - defaultTombstoneDropTimeHistogram(), + defaultTombstoneDropTimeHistogramBuilder().build(), 0, Collections.emptyList(), Collections.emptyList(), @@ -94,7 +94,7 @@ public class MetadataCollector implements PartitionStatisticsCollector protected final MinMaxIntTracker localDeletionTimeTracker = new MinMaxIntTracker(Cell.NO_DELETION_TIME, Cell.NO_DELETION_TIME); protected final MinMaxIntTracker ttlTracker = new MinMaxIntTracker(Cell.NO_TTL, Cell.NO_TTL); protected double compressionRatio = NO_COMPRESSI
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d44a0d25 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d44a0d25 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d44a0d25 Branch: refs/heads/cassandra-3.11 Commit: d44a0d25bb9a07f5e43a9951bd13f8259dfccd36 Parents: 03155b0 ef5ac1a Author: Jeff Jirsa Authored: Tue Sep 5 09:54:36 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 09:55:26 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 2 +- .../io/sstable/metadata/MetadataCollector.java | 10 +- .../cassandra/utils/StreamingHistogram.java | 333 ++- .../microbench/StreamingHistogramBench.java | 192 ++- .../cassandra/utils/StreamingHistogramTest.java | 43 +-- 6 files changed, 306 insertions(+), 275 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/CHANGES.txt -- diff --cc CHANGES.txt index 7056075,853bf61..0faed3d --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,17 -1,9 +1,18 @@@ -3.0.15 +3.11.1 + * Add a compaction option to TWCS to ignore sstables overlapping checks (CASSANDRA-13418) + * BTree.Builder memory leak (CASSANDRA-13754) + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to correctness (CASSANDRA-13798) + * Fix cassandra-stress hang issues when an error during cluster connection happens (CASSANDRA-12938) + * Better bootstrap failure message when blocked by (potential) range movement (CASSANDRA-13744) + * "ignore" option is ignored in sstableloader (CASSANDRA-13721) + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652) + * Duplicate the buffer before passing it to analyser in SASI operation (CASSANDRA-13512) + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641) +Merged from 3.0: + * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) * Fix race condition in read command serialization (CASSANDRA-13363) - * Enable segement creation before recovering commitlogs (CASSANDRA-13587) * Fix AssertionError in short read protection (CASSANDRA-13747) * Don't skip corrupted sstables on startup (CASSANDRA-13620) * Fix the merging of cells with different user type versions (CASSANDRA-13776) http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/utils/StreamingHistogram.java -- diff --cc src/java/org/apache/cassandra/utils/StreamingHistogram.java index 9114c7d,6500a1a..df49d8d --- a/src/java/org/apache/cassandra/utils/StreamingHistogram.java +++ b/src/java/org/apache/cassandra/utils/StreamingHistogram.java @@@ -24,6 -24,6 +24,7 @@@ import com.google.common.base.Objects import org.apache.cassandra.db.TypeSizes; import org.apache.cassandra.io.ISerializer; ++import org.apache.cassandra.io.sstable.SSTable; import org.apache.cassandra.io.util.DataInputPlus; import org.apache.cassandra.io.util.DataOutputPlus; @@@ -39,172 -39,15 +40,33 @@@ public class StreamingHistogra public static final StreamingHistogramSerializer serializer = new StreamingHistogramSerializer(); // TreeMap to hold bins of histogram. -private final TreeMap bin; +// The key is a numeric type so we can avoid boxing/unboxing streams of different key types +// The value is a unboxed long array always of length == 1 +// Serialized Histograms always writes with double keys for backwards compatibility +private final TreeMap bin; + - // Keep a second, larger buffer to spool data in, before finalizing it into `bin` - private final TreeMap spool; - +// maximum bin size for this histogram private final int maxBinSize; - // maximum size of the spool - private final int maxSpoolSize; - - // voluntarily give up resolution for speed - private final int roundSeconds; -private StreamingHistogram(int maxBinSize, Map bin) + +/** + * Creates a new histogram with max bin size of maxBinSize +
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/733ea88f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/733ea88f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/733ea88f Branch: refs/heads/trunk Commit: 733ea88faafa343eb6991a691f0bf36b2d5f0ed5 Parents: 33ab490 d44a0d2 Author: Jeff Jirsa Authored: Tue Sep 5 09:55:43 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 09:56:05 2017 -0700 -- CHANGES.txt | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/733ea88f/CHANGES.txt -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[3/6] cassandra git commit: StreamingHistogram is not thread safe
StreamingHistogram is not thread safe Patch by Jeff Jirsa; Reviewed by Jason Brown, Marcus Eriksson for CASSANDRA-13756 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ef5ac1a4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ef5ac1a4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ef5ac1a4 Branch: refs/heads/trunk Commit: ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2 Parents: 1e80e35 Author: Jeff Jirsa Authored: Tue Aug 29 10:31:16 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 09:54:27 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 2 +- .../io/sstable/metadata/MetadataCollector.java | 10 +- .../cassandra/utils/StreamingHistogram.java | 284 ++- .../microbench/StreamingHistogramBench.java | 192 +++-- .../cassandra/utils/StreamingHistogramTest.java | 31 +- 6 files changed, 263 insertions(+), 257 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 1ed3fe6..853bf61 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) * Fix race condition in read command serialization (CASSANDRA-13363) http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 183176c..983d6b1 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -2415,7 +2415,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean long allColumns = 0; int localTime = (int)(System.currentTimeMillis()/1000); -for (SSTableReader sstable : getSSTables(SSTableSet.LIVE)) +for (SSTableReader sstable : getSSTables(SSTableSet.CANONICAL)) { allDroppable += sstable.getDroppableTombstonesBefore(localTime - sstable.metadata.params.gcGraceSeconds); allColumns += sstable.getEstimatedColumnCount().mean() * sstable.getEstimatedColumnCount().count(); http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java index 3b13cf4..487a932 100644 --- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java +++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java @@ -59,9 +59,9 @@ public class MetadataCollector implements PartitionStatisticsCollector return new EstimatedHistogram(150); } -static StreamingHistogram defaultTombstoneDropTimeHistogram() +static StreamingHistogram.StreamingHistogramBuilder defaultTombstoneDropTimeHistogramBuilder() { -return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS); +return new StreamingHistogram.StreamingHistogramBuilder(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS); } public static StatsMetadata defaultStatsMetadata() @@ -76,7 +76,7 @@ public class MetadataCollector implements PartitionStatisticsCollector 0, Integer.MAX_VALUE, NO_COMPRESSION_RATIO, - defaultTombstoneDropTimeHistogram(), + defaultTombstoneDropTimeHistogramBuilder().build(), 0, Collections.emptyList(), Collections.emptyList(), @@ -94,7 +94,7 @@ public class MetadataCollector implements PartitionStatisticsCollector protected final MinMaxIntTracker localDeletionTimeTracker = new MinMaxIntTracker(Cell.NO_DELETION_TIME, Cell.NO_DELETION_TIME); protected final MinMaxIntTracker ttlTracker = new MinMaxIntTracker(Cell.NO_TTL, Cell.NO_TTL); protected double compressionRatio = NO_COMPRESSION_RATIO;
[1/6] cassandra git commit: StreamingHistogram is not thread safe
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 1e80e3539 -> ef5ac1a4a refs/heads/cassandra-3.11 03155b021 -> d44a0d25b refs/heads/trunk 33ab4902a -> 733ea88fa StreamingHistogram is not thread safe Patch by Jeff Jirsa; Reviewed by Jason Brown, Marcus Eriksson for CASSANDRA-13756 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ef5ac1a4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ef5ac1a4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ef5ac1a4 Branch: refs/heads/cassandra-3.0 Commit: ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2 Parents: 1e80e35 Author: Jeff Jirsa Authored: Tue Aug 29 10:31:16 2017 -0700 Committer: Jeff Jirsa Committed: Tue Sep 5 09:54:27 2017 -0700 -- CHANGES.txt | 1 + .../apache/cassandra/db/ColumnFamilyStore.java | 2 +- .../io/sstable/metadata/MetadataCollector.java | 10 +- .../cassandra/utils/StreamingHistogram.java | 284 ++- .../microbench/StreamingHistogramBench.java | 192 +++-- .../cassandra/utils/StreamingHistogramTest.java | 31 +- 6 files changed, 263 insertions(+), 257 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 1ed3fe6..853bf61 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.15 + * StreamingHistogram is not thread safe (CASSANDRA-13756) * Fix MV timestamp issues (CASSANDRA-11500) * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) * Fix race condition in read command serialization (CASSANDRA-13363) http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java index 183176c..983d6b1 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java @@ -2415,7 +2415,7 @@ public class ColumnFamilyStore implements ColumnFamilyStoreMBean long allColumns = 0; int localTime = (int)(System.currentTimeMillis()/1000); -for (SSTableReader sstable : getSSTables(SSTableSet.LIVE)) +for (SSTableReader sstable : getSSTables(SSTableSet.CANONICAL)) { allDroppable += sstable.getDroppableTombstonesBefore(localTime - sstable.metadata.params.gcGraceSeconds); allColumns += sstable.getEstimatedColumnCount().mean() * sstable.getEstimatedColumnCount().count(); http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java index 3b13cf4..487a932 100644 --- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java +++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java @@ -59,9 +59,9 @@ public class MetadataCollector implements PartitionStatisticsCollector return new EstimatedHistogram(150); } -static StreamingHistogram defaultTombstoneDropTimeHistogram() +static StreamingHistogram.StreamingHistogramBuilder defaultTombstoneDropTimeHistogramBuilder() { -return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS); +return new StreamingHistogram.StreamingHistogramBuilder(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS); } public static StatsMetadata defaultStatsMetadata() @@ -76,7 +76,7 @@ public class MetadataCollector implements PartitionStatisticsCollector 0, Integer.MAX_VALUE, NO_COMPRESSION_RATIO, - defaultTombstoneDropTimeHistogram(), + defaultTombstoneDropTimeHistogramBuilder().build(), 0, Collections.emptyList(), Collections.emptyList(), @@ -94,7 +94,7 @@ public class MetadataCollector implements PartitionStatisticsCollector protected final MinMaxIntTracker localDeletionTimeTracker = new MinMaxIntTracker(Cell.NO_DELETION
[jira] [Commented] (CASSANDRA-13818) Add support for --hosts, --force, and subrange repair to incremental repair
[ https://issues.apache.org/jira/browse/CASSANDRA-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153945#comment-16153945 ] Blake Eggleston commented on CASSANDRA-13818: - The reason I didn't go that way was because it would mean adding another repair path for a repair option that shouldn't be used very often. > Add support for --hosts, --force, and subrange repair to incremental repair > --- > > Key: CASSANDRA-13818 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13818 > Project: Cassandra > Issue Type: Improvement >Reporter: Blake Eggleston >Assignee: Blake Eggleston > Fix For: 4.0 > > > It should be possible to run incremental repair with nodes down, we just > shouldn't promote the data to repaired afterwards -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13655) Range deletes in a CAS batch are ignored
[ https://issues.apache.org/jira/browse/CASSANDRA-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147711#comment-16147711 ] Jeff Jirsa edited comment on CASSANDRA-13655 at 9/5/17 4:30 PM: Unit tests are happy. DTests have the typical flakey tests + some failures that look environmental in 3.0 and 3.11, but trunk didnt run at all. The cqlsh copy dtests failed on both 3.0 and 3.11, and they both show a single failure in the last dozen runs, so that seems odd and I'll check that on the next run. URLs will be: Edit: rebuilding again ( no tests ran ) https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/276/testReport/ https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/277/testReport/ https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/278/testReport/ was (Author: jjirsa): Unit tests are happy. DTests have the typical flakey tests + some failures that look environmental in 3.0 and 3.11, but trunk didnt run at all. The cqlsh copy dtests failed on both 3.0 and 3.11, and they both show a single failure in the last dozen runs, so that seems odd and I'll check that on the next run. URLs will be: https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/247/testReport/ https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/248/testReport/ https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/249/testReport/ > Range deletes in a CAS batch are ignored > > > Key: CASSANDRA-13655 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13655 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Blocker > Fix For: 3.0.x, 3.11.x, 4.x > > > Range deletes in a CAS batch are ignored -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13837) Hanging threads in BulkLoader
[ https://issues.apache.org/jira/browse/CASSANDRA-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153917#comment-16153917 ] Joel Knighton commented on CASSANDRA-13837: --- I pointed [~mshuler] here prematurely - my bad. It looks like the System.exit uncommented in [CASSANDRA-13836] resolves the hang we were seeing on that test. The remaining hangs seem to be repair-related and unrelated to this ticket. > Hanging threads in BulkLoader > - > > Key: CASSANDRA-13837 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13837 > Project: Cassandra > Issue Type: Bug >Reporter: Jason Brown >Assignee: Jason Brown >Priority: Minor > > [~krummas] discovered some threads that were not closing correctly when he > fixed CASSANDRA-13836. We suspect this is due to > CASSANDRA-8457/CASSANDRA-12229. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[ https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-13844: Reproduced In: 2.2.10, 2.1.18 (was: 2.1.18, 2.2.10) Reviewer: Alex Petrov > sstableloader doesn't support non default storage_port and ssl_storage_port > --- > > Key: CASSANDRA-13844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 2.2.11, 2.1.19 > > > Currently *storage_port* and *ssl_storage_port* are hardcoded to the > defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so > this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13530) GroupCommitLogService
[ https://issues.apache.org/jira/browse/CASSANDRA-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153918#comment-16153918 ] Ariel Weisberg commented on CASSANDRA-13530: That documentation in the YAML looks wrong to me. There is no default value it requires you to supply one apparently. There is a [common bit of functionality|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogService.java#L26] with both periodic and batch commit log where it will run syncs periodically and that is in the shared superclass. Then batch itself [requests an extra sync|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogService.java#L33] every time it accepts a write so the periodic one does nothing productive as far as I can tell. Two seconds should be fine. The periodic one will never fire because every time the sync thread wakes up it recalculates the next wakeup time so it will never reach the periodic wakeup. > GroupCommitLogService > - > > Key: CASSANDRA-13530 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13530 > Project: Cassandra > Issue Type: Improvement >Reporter: Yuji Ito >Assignee: Yuji Ito > Fix For: 2.2.x, 3.0.x, 3.11.x > > Attachments: groupCommit22.patch, groupCommit30.patch, > groupCommit3x.patch, groupCommitLog_noSerial_result.xlsx, > groupCommitLog_result.xlsx, GuavaRequestThread.java, MicroRequestThread.java > > > I propose a new CommitLogService, GroupCommitLogService, to improve the > throughput when lots of requests are received. > It improved the throughput by maximum 94%. > I'd like to discuss about this CommitLogService. > Currently, we can select either 2 CommitLog services; Periodic and Batch. > In Periodic, we might lose some commit log which hasn't written to the disk. > In Batch, we can write commit log to the disk every time. The size of commit > log to write is too small (< 4KB). When high concurrency, these writes are > gathered and persisted to the disk at once. But, when insufficient > concurrency, many small writes are issued and the performance decreases due > to the latency of the disk. Even if you use SSD, processes of many IO > commands decrease the performance. > GroupCommitLogService writes some commitlog to the disk at once. > The patch adds GroupCommitLogService (It is enabled by setting > `commitlog_sync` and `commitlog_sync_group_window_in_ms` in cassandra.yaml). > The difference from Batch is just only waiting for the semaphore. > By waiting for the semaphore, some writes for commit logs are executed at the > same time. > In GroupCommitLogService, the latency becomes worse if the there is no > concurrency. > I measured the performance with my microbench (MicroRequestThread.java) by > increasing the number of threads.The cluster has 3 nodes (Replication factor: > 3). Each nodes is AWS EC2 m4.large instance + 200IOPS io1 volume. > The result is as below. The GroupCommitLogService with 10ms window improved > update with Paxos by 94% and improved select with Paxos by 76%. > h6. SELECT / sec > ||\# of threads||Batch 2ms||Group 10ms|| > |1|192|103| > |2|163|212| > |4|264|416| > |8|454|800| > |16|744|1311| > |32|1151|1481| > |64|1767|1844| > |128|2949|3011| > |256|4723|5000| > h6. UPDATE / sec > ||\# of threads||Batch 2ms||Group 10ms|| > |1|45|26| > |2|39|51| > |4|58|102| > |8|102|198| > |16|167|213| > |32|289|295| > |64|544|548| > |128|1046|1058| > |256|2020|2061| -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[ https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-13844: Reproduced In: 2.2.10, 2.1.18 Status: Patch Available (was: In Progress) > sstableloader doesn't support non default storage_port and ssl_storage_port > --- > > Key: CASSANDRA-13844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 2.2.11, 2.1.19 > > > Currently *storage_port* and *ssl_storage_port* are hardcoded to the > defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so > this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[ https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153893#comment-16153893 ] Eduard Tudenhoefner edited comment on CASSANDRA-13844 at 9/5/17 4:21 PM: - 2.1 Branch: https://github.com/nastra/cassandra/tree/13844-21 2.1 Test: https://circleci.com/gh/nastra/cassandra/18 2.2 Branch: https://github.com/nastra/cassandra/tree/13844-22 2.2 Test: https://circleci.com/gh/nastra/cassandra/19 2.2 upstream merges should be all empty was (Author: eduard.tudenhoefner): Branch: https://github.com/nastra/cassandra/tree/13844-21 Test: https://circleci.com/gh/nastra/cassandra/17 > sstableloader doesn't support non default storage_port and ssl_storage_port > --- > > Key: CASSANDRA-13844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 2.2.11, 2.1.19 > > > Currently *storage_port* and *ssl_storage_port* are hardcoded to the > defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so > this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[ https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153893#comment-16153893 ] Eduard Tudenhoefner commented on CASSANDRA-13844: - Branch: https://github.com/nastra/cassandra/tree/13844-21 Test: https://circleci.com/gh/nastra/cassandra/17 > sstableloader doesn't support non default storage_port and ssl_storage_port > --- > > Key: CASSANDRA-13844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 2.2.11, 2.1.19 > > > Currently *storage_port* and *ssl_storage_port* are hardcoded to the > defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so > this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
Eduard Tudenhoefner created CASSANDRA-13844: --- Summary: sstableloader doesn't support non default storage_port and ssl_storage_port Key: CASSANDRA-13844 URL: https://issues.apache.org/jira/browse/CASSANDRA-13844 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Eduard Tudenhoefner Assignee: Eduard Tudenhoefner Fix For: 2.2.11, 2.1.19 Currently *storage_port* and *ssl_storage_port* are hardcoded to the defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so this here is just backporting it to C* 2.1/2.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13837) Hanging threads in BulkLoader
[ https://issues.apache.org/jira/browse/CASSANDRA-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153868#comment-16153868 ] Michael Shuler commented on CASSANDRA-13837: This appears to be hanging up the trunk dtest jobs in random tests. We've been looking for a single dtest that might be problematic, but there doesn't seem to be a clear pattern on a particular test. Any way we can bump the priority on this from Minor? > Hanging threads in BulkLoader > - > > Key: CASSANDRA-13837 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13837 > Project: Cassandra > Issue Type: Bug >Reporter: Jason Brown >Assignee: Jason Brown >Priority: Minor > > [~krummas] discovered some threads that were not closing correctly when he > fixed CASSANDRA-13836. We suspect this is due to > CASSANDRA-8457/CASSANDRA-12229. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR
[ https://issues.apache.org/jira/browse/CASSANDRA-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Jorgensen updated CASSANDRA-13843: - Labels: newbie patch (was: ) Fix Version/s: 4.x 3.11.x Reproduced In: 3.11.x, 4.x Status: Patch Available (was: Open) > Debian init shadows CASSANDRA_HEAPDUMP_DIR > -- > > Key: CASSANDRA-13843 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13843 > Project: Cassandra > Issue Type: Bug > Components: Observability >Reporter: Andrew Jorgensen >Priority: Minor > Labels: newbie, patch > Fix For: 3.11.x, 4.x > > Attachments: 0001-Remove-debian-init-setting-heap-dump-file.patch > > > The debian init script sets the heap dump file directly using the cassandra > users home directory and the -H flag to the cassandra > process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76]. > The cassandra heap dump location can also be set in the cassandra-env.sh > file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump > location is based off the home directory of the cassandra user and cannot > easily be changed. Also unfortunately if you do `ps aux | grep casandra` you > can clearly see that the -H flag takes precedent over the value found in > cassandra-env.sh. This makes it difficult to change the heap dump location > for cassandra and is non-intuitive when the value is set in cassandra-env.sh > why the heap dump does not actually end up in the correct place. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR
[ https://issues.apache.org/jira/browse/CASSANDRA-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Jorgensen updated CASSANDRA-13843: - Attachment: 0001-Remove-debian-init-setting-heap-dump-file.patch > Debian init shadows CASSANDRA_HEAPDUMP_DIR > -- > > Key: CASSANDRA-13843 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13843 > Project: Cassandra > Issue Type: Bug > Components: Observability >Reporter: Andrew Jorgensen >Priority: Minor > Attachments: 0001-Remove-debian-init-setting-heap-dump-file.patch > > > The debian init script sets the heap dump file directly using the cassandra > users home directory and the -H flag to the cassandra > process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76]. > The cassandra heap dump location can also be set in the cassandra-env.sh > file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump > location is based off the home directory of the cassandra user and cannot > easily be changed. Also unfortunately if you do `ps aux | grep casandra` you > can clearly see that the -H flag takes precedent over the value found in > cassandra-env.sh. This makes it difficult to change the heap dump location > for cassandra and is non-intuitive when the value is set in cassandra-env.sh > why the heap dump does not actually end up in the correct place. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13662) Remove unsupported CREDENTIALS message
[ https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13662: --- Resolution: Fixed Fix Version/s: (was: 4.x) 4.0 Status: Resolved (was: Ready to Commit) Thanks Jeremiah! Merged as 33ab4902a. > Remove unsupported CREDENTIALS message > -- > > Key: CASSANDRA-13662 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13662 > Project: Cassandra > Issue Type: Improvement > Components: Auth, CQL >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Minor > Labels: security > Fix For: 4.0 > > > Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try > not to keep unused legacy classes around for any security relevant features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Remove obsoleted CredentialsMessage
Repository: cassandra Updated Branches: refs/heads/trunk c1f623902 -> 33ab4902a Remove obsoleted CredentialsMessage patch by Stefan Podkowinski; reviewed by Jeremiah Jordan for CASSANDRA-13662 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/33ab4902 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/33ab4902 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/33ab4902 Branch: refs/heads/trunk Commit: 33ab4902a1bef5aa662b80b89b5dd9a318f67db5 Parents: c1f6239 Author: Stefan Podkowinski Authored: Tue Jul 4 10:05:51 2017 +0200 Committer: Stefan Podkowinski Committed: Tue Sep 5 16:11:42 2017 +0200 -- .../org/apache/cassandra/transport/Client.java | 7 -- .../org/apache/cassandra/transport/Frame.java | 37 +--- .../org/apache/cassandra/transport/Message.java | 2 +- .../cassandra/transport/SimpleClient.java | 8 -- .../transport/messages/CredentialsMessage.java | 97 .../messages/UnsupportedMessageCodec.java | 56 +++ .../cassandra/transport/ProtocolErrorTest.java | 21 + 7 files changed, 102 insertions(+), 126 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/33ab4902/src/java/org/apache/cassandra/transport/Client.java -- diff --git a/src/java/org/apache/cassandra/transport/Client.java b/src/java/org/apache/cassandra/transport/Client.java index 9a76e03..7fec473 100644 --- a/src/java/org/apache/cassandra/transport/Client.java +++ b/src/java/org/apache/cassandra/transport/Client.java @@ -175,13 +175,6 @@ public class Client extends SimpleClient { return new OptionsMessage(); } -else if (msgType.equals("CREDENTIALS")) -{ -System.err.println("[WARN] CREDENTIALS command is deprecated, use AUTHENTICATE instead"); -CredentialsMessage msg = new CredentialsMessage(); -msg.credentials.putAll(readCredentials(iter)); -return msg; -} else if (msgType.equals("AUTHENTICATE")) { Map credentials = readCredentials(iter); http://git-wip-us.apache.org/repos/asf/cassandra/blob/33ab4902/src/java/org/apache/cassandra/transport/Frame.java -- diff --git a/src/java/org/apache/cassandra/transport/Frame.java b/src/java/org/apache/cassandra/transport/Frame.java index 6cd8b1e..41e64f9 100644 --- a/src/java/org/apache/cassandra/transport/Frame.java +++ b/src/java/org/apache/cassandra/transport/Frame.java @@ -22,6 +22,8 @@ import java.io.IOException; import java.util.EnumSet; import java.util.List; +import com.google.common.annotations.VisibleForTesting; + import io.netty.buffer.ByteBuf; import io.netty.channel.*; import io.netty.handler.codec.ByteToMessageDecoder; @@ -146,8 +148,8 @@ public class Frame this.factory = factory; } -@Override -protected void decode(ChannelHandlerContext ctx, ByteBuf buffer, List results) +@VisibleForTesting +Frame decodeFrame(ByteBuf buffer) throws Exception { if (discardingTooLongFrame) @@ -156,12 +158,12 @@ public class Frame // If we have discarded everything, throw the exception if (bytesToDiscard <= 0) fail(); -return; +return null; } int readableBytes = buffer.readableBytes(); if (readableBytes == 0) -return; +return null; int idx = buffer.readerIndex(); @@ -174,7 +176,7 @@ public class Frame // Wait until we have the complete header if (readableBytes < Header.LENGTH) -return; +return null; int flags = buffer.getByte(idx++); EnumSet decodedFlags = Header.Flag.deserialize(flags); @@ -210,11 +212,11 @@ public class Frame bytesToDiscard = discard(buffer, frameLength); if (bytesToDiscard <= 0) fail(); -return; +return null; } if (buffer.readableBytes() < frameLength) -return; +return null; // extract body ByteBuf body = buffer.slice(idx, (int) bodyLength); @@ -223,24 +225,33 @@ public class Frame idx += bodyLength; buffer.readerIndex(idx); +return new Frame(new Header(version, decodedFlags, streamId, type), body); +} + +@Override +protected void decode(ChannelHandlerContext ctx, ByteBuf buffer, List results) +throws Exception +
[jira] [Commented] (CASSANDRA-13662) Remove unsupported CREDENTIALS message
[ https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153700#comment-16153700 ] Jeremiah Jordan commented on CASSANDRA-13662: - https://github.com/spodkowinski/cassandra/commit/c23d00808ac37d5d6f71810b17c7da2a2e73831b looks good to me. > Remove unsupported CREDENTIALS message > -- > > Key: CASSANDRA-13662 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13662 > Project: Cassandra > Issue Type: Improvement > Components: Auth, CQL >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Minor > Labels: security > Fix For: 4.x > > > Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try > not to keep unused legacy classes around for any security relevant features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13703) Using min_compress_ratio <= 1 causes corruption
[ https://issues.apache.org/jira/browse/CASSANDRA-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153675#comment-16153675 ] Dimitar Dimitrov edited comment on CASSANDRA-13703 at 9/5/17 1:56 PM: -- +1 - from my still largely layman perspective, the change looks good. A couple of small nits: * {{CompressedChunkReader.maybeCheckCrc()}} (renamed in this patch to {{shouldCheckCrc()}}) may be removed or at least its visibility could be reduced. It seems to be used solely in CompressedChunkReader.java, lines 158 and 204 in this patch. * The no-argument / single-argument factory methods for Snappy and LZ4 compressions in CompressionParams.java seem to differ in the values that they default to for min compression ratio and max compressed length. ** For Snappy, if nothing is specified, or only chunk length is specified, a default min compression ratio of 1.1 is used, and therefore max compressed length ends up somewhere roughly around 90% of chunk length. ** For LZ4, if nothing is specified, or only chunk length is specified, a default max compressed length of chunk length is used, and therefore min compression ratio ends up at 1.0 (I'm not sure if a precision error is possible there). Edit: Of course, take this review with the appropriate rock-sized grain of salt. was (Author: dimitarndimitrov): +1 - from my still largely layman perspective, the change looks good. A couple of small nits: * {{CompressedChunkReader.maybeCheckCrc()}} (renamed in this patch to {{shouldCheckCrc()}}) may be removed or at least its visibility could be reduced. It seems to be used solely in CompressedChunkReader.java, lines 158 and 204 in this patch. * The no-argument / single-argument factory methods for Snappy and LZ4 compressions in CompressionParams.java seem to differ in the values that they default to for min compression ratio and max compressed length. ** For Snappy, if nothing is specified, or only chunk length is specified, a default min compression ratio of 1.1 is used, and therefore max compressed length ends up somewhere roughly around 90% of chunk length. ** For LZ4, if nothing is specified, or only chunk length is specified, a default max compressed length of chunk length is used, and therefore min compression ratio ends up at 1.0 (I'm not sure if a precision error is possible there). > Using min_compress_ratio <= 1 causes corruption > --- > > Key: CASSANDRA-13703 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13703 > Project: Cassandra > Issue Type: Bug >Reporter: Branimir Lambov >Assignee: Branimir Lambov >Priority: Blocker > Fix For: 4.x > > Attachments: patch > > > This is because chunks written uncompressed end up below the compressed size > threshold. Demonstrated by applying the attached patch meant to improve the > testing of the 10520 changes, and running > {{CompressedSequentialWriterTest.testLZ4Writer}}. > The default {{min_compress_ratio: 0}} is not affected as it never writes > uncompressed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13703) Using min_compress_ratio <= 1 causes corruption
[ https://issues.apache.org/jira/browse/CASSANDRA-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153675#comment-16153675 ] Dimitar Dimitrov commented on CASSANDRA-13703: -- +1 - from my still largely layman perspective, the change looks good. A couple of small nits: * {{CompressedChunkReader.maybeCheckCrc()}} (renamed in this patch to {{shouldCheckCrc()}}) may be removed or at least its visibility could be reduced. It seems to be used solely in CompressedChunkReader.java, lines 158 and 204 in this patch. * The no-argument / single-argument factory methods for Snappy and LZ4 compressions in CompressionParams.java seem to differ in the values that they default to for min compression ratio and max compressed length. ** For Snappy, if nothing is specified, or only chunk length is specified, a default min compression ratio of 1.1 is used, and therefore max compressed length ends up somewhere roughly around 90% of chunk length. ** For LZ4, if nothing is specified, or only chunk length is specified, a default max compressed length of chunk length is used, and therefore min compression ratio ends up at 1.0 (I'm not sure if a precision error is possible there). > Using min_compress_ratio <= 1 causes corruption > --- > > Key: CASSANDRA-13703 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13703 > Project: Cassandra > Issue Type: Bug >Reporter: Branimir Lambov >Assignee: Branimir Lambov >Priority: Blocker > Fix For: 4.x > > Attachments: patch > > > This is because chunks written uncompressed end up below the compressed size > threshold. Demonstrated by applying the attached patch meant to improve the > testing of the 10520 changes, and running > {{CompressedSequentialWriterTest.testLZ4Writer}}. > The default {{min_compress_ratio: 0}} is not affected as it never writes > uncompressed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR
[ https://issues.apache.org/jira/browse/CASSANDRA-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Jorgensen updated CASSANDRA-13843: - Description: The debian init script sets the heap dump file directly using the cassandra users home directory and the -H flag to the cassandra process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76]. The cassandra heap dump location can also be set in the cassandra-env.sh file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump location is based off the home directory of the cassandra user and cannot easily be changed. Also unfortunately if you do `ps aux | grep casandra` you can clearly see that the -H flag takes precedent over the value found in cassandra-env.sh. This makes it difficult to change the heap dump location for cassandra and is non-intuitive when the value is set in cassandra-env.sh why the heap dump does not actually end up in the correct place. (was: The debian init script sets the heap dump file directly using the cassandra users home directory and the -H flag to the cassandra process[1 title|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76]. The cassandra heap dump location can also be set in the cassandra-env.sh file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump location is based off the home directory of the cassandra user and cannot easily be changed. Also unfortunately if you do `ps aux | grep casandra` you can clearly see that the -H flag takes precedent over the value found in cassandra-env.sh. This makes it difficult to change the heap dump location for cassandra and is non-intuitive when the value is set in cassandra-env.sh why the heap dump does not actually end up in the correct place.) > Debian init shadows CASSANDRA_HEAPDUMP_DIR > -- > > Key: CASSANDRA-13843 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13843 > Project: Cassandra > Issue Type: Bug > Components: Observability >Reporter: Andrew Jorgensen >Priority: Minor > > The debian init script sets the heap dump file directly using the cassandra > users home directory and the -H flag to the cassandra > process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76]. > The cassandra heap dump location can also be set in the cassandra-env.sh > file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump > location is based off the home directory of the cassandra user and cannot > easily be changed. Also unfortunately if you do `ps aux | grep casandra` you > can clearly see that the -H flag takes precedent over the value found in > cassandra-env.sh. This makes it difficult to change the heap dump location > for cassandra and is non-intuitive when the value is set in cassandra-env.sh > why the heap dump does not actually end up in the correct place. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR
Andrew Jorgensen created CASSANDRA-13843: Summary: Debian init shadows CASSANDRA_HEAPDUMP_DIR Key: CASSANDRA-13843 URL: https://issues.apache.org/jira/browse/CASSANDRA-13843 Project: Cassandra Issue Type: Bug Components: Observability Reporter: Andrew Jorgensen Priority: Minor The debian init script sets the heap dump file directly using the cassandra users home directory and the -H flag to the cassandra process[1 title|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76]. The cassandra heap dump location can also be set in the cassandra-env.sh file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump location is based off the home directory of the cassandra user and cannot easily be changed. Also unfortunately if you do `ps aux | grep casandra` you can clearly see that the -H flag takes precedent over the value found in cassandra-env.sh. This makes it difficult to change the heap dump location for cassandra and is non-intuitive when the value is set in cassandra-env.sh why the heap dump does not actually end up in the correct place. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12696) Allow to change logging levels based on components
[ https://issues.apache.org/jira/browse/CASSANDRA-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-12696: --- Resolution: Fixed Reviewer: Chris Lohfink Status: Resolved (was: Ready to Commit) Thanks! Merged as c1f6239. I don't think keeping the list of classes in sync will be that much of an issue in practice. I've went through the list again that I've created a year ago and all classes still exist and seem to be still relevant. But patching the list should not be an issue in case something's missing. > Allow to change logging levels based on components > -- > > Key: CASSANDRA-12696 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12696 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Minor > Labels: lhf > Attachments: 12696-trunk.patch > > > Currently users are able to dynamically change logging configuration by using > {{nodetool setlogginglevel }}. Unfortunately this requires to > know a bit about the Cassandra package hierarchy and gathering all the > involved packages/classes can be tedious, especially in troubleshooting > situations. What I'd like to have is a way to tell a user to "_when X > happens, enable debug logs for bootstrapping/repair/compactions/.._" by > simply running e.g. {{nodetool setlogginglevel bootstrap DEBUG}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Allow to change logging levels based on components
Repository: cassandra Updated Branches: refs/heads/trunk e38015ed1 -> c1f623902 Allow to change logging levels based on components patch by Stefan Podkowinski; reviewed by Chris Lohfink for CASSANDRA-12696 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c1f62390 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c1f62390 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c1f62390 Branch: refs/heads/trunk Commit: c1f62390239f792f010feb07b02218062e34cee0 Parents: e38015e Author: Stefan Podkowinski Authored: Fri Sep 23 10:56:43 2016 +0200 Committer: Stefan Podkowinski Committed: Tue Sep 5 15:16:40 2017 +0200 -- CHANGES.txt | 1 + .../tools/nodetool/SetLoggingLevel.java | 68 ++-- 2 files changed, 65 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c1f62390/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 845fced..3fd8ab0 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0 + * Allow changing log levels via nodetool for related classes (CASSANDRA-12696) * Add stress profile yaml with LWT (CASSANDRA-7960) * Reduce memory copies and object creations when acting on ByteBufs (CASSANDRA-13789) * simplify mx4j configuration (Cassandra-13578) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c1f62390/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java -- diff --git a/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java b/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java index 94e4e3b..8d9ad90 100644 --- a/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java +++ b/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java @@ -22,22 +22,82 @@ import io.airlift.airline.Arguments; import io.airlift.airline.Command; import java.util.ArrayList; +import java.util.Collections; import java.util.List; +import com.google.common.collect.Lists; + import org.apache.cassandra.tools.NodeProbe; import org.apache.cassandra.tools.NodeTool.NodeToolCmd; -@Command(name = "setlogginglevel", description = "Set the log level threshold for a given class. If both class and level are empty/null, it will reset to the initial configuration") +@Command(name = "setlogginglevel", description = "Set the log level threshold for a given component or class. Will reset to the initial configuration if called with no parameters.") public class SetLoggingLevel extends NodeToolCmd { -@Arguments(usage = " ", description = "The class to change the level for and the log level threshold to set (can be empty)") +@Arguments(usage = " ", description = "The component or class to change the level for and the log level threshold to set. Will reset to initial level if omitted. " ++ "Available components: bootstrap, compaction, repair, streaming, cql, ring") private List args = new ArrayList<>(); @Override public void execute(NodeProbe probe) { -String classQualifier = args.size() >= 1 ? args.get(0) : EMPTY; +String target = args.size() >= 1 ? args.get(0) : EMPTY; String level = args.size() == 2 ? args.get(1) : EMPTY; -probe.setLoggingLevel(classQualifier, level); + +List classQualifiers = Collections.singletonList(target); +if (target.equals("bootstrap")) +{ +classQualifiers = Lists.newArrayList( +"org.apache.cassandra.gms", +"org.apache.cassandra.hints", +"org.apache.cassandra.schema", +"org.apache.cassandra.service.StorageService", +"org.apache.cassandra.db.SystemKeyspace", +"org.apache.cassandra.batchlog.BatchlogManager", +"org.apache.cassandra.net.MessagingService"); +} +else if (target.equals("repair")) +{ +classQualifiers = Lists.newArrayList( +"org.apache.cassandra.repair", +"org.apache.cassandra.db.compaction.CompactionManager", +"org.apache.cassandra.service.SnapshotVerbHandler"); +} +else if (target.equals("streaming")) +{ +classQualifiers = Lists.newArrayList( +"org.apache.cassandra.streaming", +"org.apache.cassandra.dht.RangeStreamer"); +} +else if (target.equals("compaction")) +{ +classQualifiers = Lists.newArrayList( +"org.apache.cassandra.db.compaction", +"org.apache.cassandra.d
[jira] [Commented] (CASSANDRA-13787) RangeTombstoneMarker and ParitionDeletion is not properly included in MV
[ https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153603#comment-16153603 ] ZhaoYang commented on CASSANDRA-13787: -- Thanks for reviewing. I will prepare branches for 3.0/3.1/trunk. I have removed the code for second issue since 11500 is merged.. > RangeTombstoneMarker and ParitionDeletion is not properly included in MV > > > Key: CASSANDRA-13787 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13787 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: ZhaoYang >Assignee: ZhaoYang > > Found two problems related to MV tombstone. > 1. Range-tombstone-Marker being ignored after shadowing first row, subsequent > base rows are not shadowed in TableViews. > If the range tombstone was not flushed, it was used as deleted row to > shadow new updates. It works correctly. > After range tombstone was flushed, it was used as RangeTombstoneMarker > and being skipped after shadowing first update. The bound of > RangeTombstoneMarker seems wrong, it contained full clustering, but it should > contain range or it should be multiple RangeTombstoneMarkers for multiple > slices(aka. new updates) > -2. Partition tombstone is not used when no existing live data, it will > resurrect deleted cells. It was found in 11500 and included in that patch.- > (Merged in CASSANDRA-11500) > In order not to make 11500 patch more complicated, I will try fix > range/partition tombstone issue here. > {code:title=Tests to reproduce} > @Test > public void testExistingRangeTombstoneWithFlush() throws Throwable > { > testExistingRangeTombstone(true); > } > @Test > public void testExistingRangeTombstoneWithoutFlush() throws Throwable > { > testExistingRangeTombstone(false); > } > public void testExistingRangeTombstone(boolean flush) throws Throwable > { > createTable("CREATE TABLE %s (k1 int, c1 int, c2 int, v1 int, v2 int, > PRIMARY KEY (k1, c1, c2))"); > execute("USE " + keyspace()); > executeNet(protocolVersion, "USE " + keyspace()); > createView("view1", >"CREATE MATERIALIZED VIEW view1 AS SELECT * FROM %%s WHERE > k1 IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL PRIMARY KEY (k1, c2, > c1)"); > updateView("DELETE FROM %s USING TIMESTAMP 10 WHERE k1 = 1 and c1=1"); > if (flush) > > Keyspace.open(keyspace()).getColumnFamilyStore(currentTable()).forceBlockingFlush(); > String table = KEYSPACE + "." + currentTable(); > updateView("BEGIN BATCH " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, > 0, 0, 0) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, > 1, 0, 1) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 0, 1, 0) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 1, 1, 1) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 2, 1, 2) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 3, 1, 3) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 2, > 0, 2, 0) USING TIMESTAMP 5; " + > "APPLY BATCH"); > assertRowsIgnoringOrder(execute("select * from %s"), > row(1, 0, 0, 0, 0), > row(1, 0, 1, 0, 1), > row(1, 2, 0, 2, 0)); > assertRowsIgnoringOrder(execute("select k1,c1,c2,v1,v2 from view1"), > row(1, 0, 0, 0, 0), > row(1, 0, 1, 0, 1), > row(1, 2, 0, 2, 0)); > } > @Test > public void testExistingParitionDeletionWithFlush() throws Throwable > { > testExistingParitionDeletion(true); > } > @Test > public void testExistingParitionDeletionWithoutFlush() throws Throwable > { > testExistingParitionDeletion(false); > } > public void testExistingParitionDeletion(boolean flush) throws Throwable > { > // for partition range deletion, need to know that existing row is > shadowed instead of not existed. > createTable("CREATE TABLE %s (a int, b int, c int, d int, PRIMARY KEY > (a))"); > execute("USE " + keyspace()); > executeNet(protocolVersion, "USE " + keyspace()); > createView("mv_test1", >"CREATE MATERIALIZED VIEW %s AS SELECT * FROM %%s WH
[jira] [Commented] (CASSANDRA-13787) RangeTombstoneMarker and ParitionDeletion is not properly included in MV
[ https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153596#comment-16153596 ] Sylvain Lebresne commented on CASSANDRA-13787: -- Sorry for the delay reviewing, but patch lgtm (though, to make extract sure we're on the same page, this appear to only include the fix for the "first problem" since the "second problem" seems to have been included in CASSANDRA-11500). Have you run CI on this? Also, if you could provide branches for all branches, that would be amazing (the fix itself should merge up cleanly, but I suspect the tests may require a few minor updates). > RangeTombstoneMarker and ParitionDeletion is not properly included in MV > > > Key: CASSANDRA-13787 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13787 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: ZhaoYang >Assignee: ZhaoYang > > Found two problems related to MV tombstone. > 1. Range-tombstone-Marker being ignored after shadowing first row, subsequent > base rows are not shadowed in TableViews. > If the range tombstone was not flushed, it was used as deleted row to > shadow new updates. It works correctly. > After range tombstone was flushed, it was used as RangeTombstoneMarker > and being skipped after shadowing first update. The bound of > RangeTombstoneMarker seems wrong, it contained full clustering, but it should > contain range or it should be multiple RangeTombstoneMarkers for multiple > slices(aka. new updates) > -2. Partition tombstone is not used when no existing live data, it will > resurrect deleted cells. It was found in 11500 and included in that patch.- > (Merged in CASSANDRA-11500) > In order not to make 11500 patch more complicated, I will try fix > range/partition tombstone issue here. > {code:title=Tests to reproduce} > @Test > public void testExistingRangeTombstoneWithFlush() throws Throwable > { > testExistingRangeTombstone(true); > } > @Test > public void testExistingRangeTombstoneWithoutFlush() throws Throwable > { > testExistingRangeTombstone(false); > } > public void testExistingRangeTombstone(boolean flush) throws Throwable > { > createTable("CREATE TABLE %s (k1 int, c1 int, c2 int, v1 int, v2 int, > PRIMARY KEY (k1, c1, c2))"); > execute("USE " + keyspace()); > executeNet(protocolVersion, "USE " + keyspace()); > createView("view1", >"CREATE MATERIALIZED VIEW view1 AS SELECT * FROM %%s WHERE > k1 IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL PRIMARY KEY (k1, c2, > c1)"); > updateView("DELETE FROM %s USING TIMESTAMP 10 WHERE k1 = 1 and c1=1"); > if (flush) > > Keyspace.open(keyspace()).getColumnFamilyStore(currentTable()).forceBlockingFlush(); > String table = KEYSPACE + "." + currentTable(); > updateView("BEGIN BATCH " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, > 0, 0, 0) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, > 1, 0, 1) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 0, 1, 0) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 1, 1, 1) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 2, 1, 2) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, > 3, 1, 3) USING TIMESTAMP 5; " + > "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 2, > 0, 2, 0) USING TIMESTAMP 5; " + > "APPLY BATCH"); > assertRowsIgnoringOrder(execute("select * from %s"), > row(1, 0, 0, 0, 0), > row(1, 0, 1, 0, 1), > row(1, 2, 0, 2, 0)); > assertRowsIgnoringOrder(execute("select k1,c1,c2,v1,v2 from view1"), > row(1, 0, 0, 0, 0), > row(1, 0, 1, 0, 1), > row(1, 2, 0, 2, 0)); > } > @Test > public void testExistingParitionDeletionWithFlush() throws Throwable > { > testExistingParitionDeletion(true); > } > @Test > public void testExistingParitionDeletionWithoutFlush() throws Throwable > { > testExistingParitionDeletion(false); > } > public void testExistingParitionDeletion(boolean flush) throws Throwable > { > // for partition range deletion, need to know that existing row is > shad
[jira] [Comment Edited] (CASSANDRA-13831) NettyFactoryTest is failing in trunk on MacOS
[ https://issues.apache.org/jira/browse/CASSANDRA-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153542#comment-16153542 ] Jason Brown edited comment on CASSANDRA-13831 at 9/5/17 12:10 PM: -- rebased and pushed a change that adds an additional check to see if the OS is linux, and only then log that epoll is unavailable. UPDATE: just saw your comment about {{NativeTransportService}}. Will move the logging there, from {{NettyFactory}} was (Author: jasobrown): rebased and pushed a change that adds an additional check to see if the OS is linux, and only then log that epoll is unavailable. > NettyFactoryTest is failing in trunk on MacOS > - > > Key: CASSANDRA-13831 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13831 > Project: Cassandra > Issue Type: Bug > Components: Testing >Reporter: Aleksey Yeschenko >Assignee: Jason Brown >Priority: Minor > Fix For: 4.x > > > Example failure: > {code} > [junit] Testcase: > getEventLoopGroup_EpollWithoutIoRatioBoost(org.apache.cassandra.net.async.NettyFactoryTest): > Caused an ERROR > [junit] failed to load the required native library > [junit] java.lang.UnsatisfiedLinkError: failed to load the required > native library > [junit] at > io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78) > [junit] at > io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:53) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47) > [junit] at > io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68) > [junit] at > org.apache.cassandra.net.async.NettyFactory.getEventLoopGroup(NettyFactory.java:175) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Epoll(NettyFactoryTest.java:187) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_EpollWithoutIoRatioBoost(NettyFactoryTest.java:205) > [junit] Caused by: java.lang.ExceptionInInitializerError > [junit] at io.netty.channel.epoll.Epoll.(Epoll.java:33) > [junit] at > org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162) > [junit] at > org.apache.cassandra.net.async.NettyFactory.(NettyFactory.java:94) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Nio(NettyFactoryTest.java:216) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_NioWithoutIoRatioBoost(NettyFactoryTest.java:211) > [junit] Caused by: java.lang.IllegalStateException: Only supported on > Linux > [junit] at > io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189) > [junit] at io.netty.channel.epoll.Native.(Native.java:61) > {code} > It's obviously caused by epoll being unavailable on MacOS, but the tests > should still be made to pass. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13831) NettyFactoryTest is failing in trunk on MacOS
[ https://issues.apache.org/jira/browse/CASSANDRA-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153542#comment-16153542 ] Jason Brown commented on CASSANDRA-13831: - rebased and pushed a change that adds an additional check to see if the OS is linux, and only then log that epoll is unavailable. > NettyFactoryTest is failing in trunk on MacOS > - > > Key: CASSANDRA-13831 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13831 > Project: Cassandra > Issue Type: Bug > Components: Testing >Reporter: Aleksey Yeschenko >Assignee: Jason Brown >Priority: Minor > Fix For: 4.x > > > Example failure: > {code} > [junit] Testcase: > getEventLoopGroup_EpollWithoutIoRatioBoost(org.apache.cassandra.net.async.NettyFactoryTest): > Caused an ERROR > [junit] failed to load the required native library > [junit] java.lang.UnsatisfiedLinkError: failed to load the required > native library > [junit] at > io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78) > [junit] at > io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:53) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47) > [junit] at > io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68) > [junit] at > org.apache.cassandra.net.async.NettyFactory.getEventLoopGroup(NettyFactory.java:175) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Epoll(NettyFactoryTest.java:187) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_EpollWithoutIoRatioBoost(NettyFactoryTest.java:205) > [junit] Caused by: java.lang.ExceptionInInitializerError > [junit] at io.netty.channel.epoll.Epoll.(Epoll.java:33) > [junit] at > org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162) > [junit] at > org.apache.cassandra.net.async.NettyFactory.(NettyFactory.java:94) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Nio(NettyFactoryTest.java:216) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_NioWithoutIoRatioBoost(NettyFactoryTest.java:211) > [junit] Caused by: java.lang.IllegalStateException: Only supported on > Linux > [junit] at > io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189) > [junit] at io.netty.channel.epoll.Native.(Native.java:61) > {code} > It's obviously caused by epoll being unavailable on MacOS, but the tests > should still be made to pass. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13831) NettyFactoryTest is failing in trunk on MacOS
[ https://issues.apache.org/jira/browse/CASSANDRA-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153534#comment-16153534 ] Jason Brown commented on CASSANDRA-13831: - Printing the error is useful on linux to know why epoll might not be available, but I agree it's not necessary on other platforms. Fixing. > NettyFactoryTest is failing in trunk on MacOS > - > > Key: CASSANDRA-13831 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13831 > Project: Cassandra > Issue Type: Bug > Components: Testing >Reporter: Aleksey Yeschenko >Assignee: Jason Brown >Priority: Minor > Fix For: 4.x > > > Example failure: > {code} > [junit] Testcase: > getEventLoopGroup_EpollWithoutIoRatioBoost(org.apache.cassandra.net.async.NettyFactoryTest): > Caused an ERROR > [junit] failed to load the required native library > [junit] java.lang.UnsatisfiedLinkError: failed to load the required > native library > [junit] at > io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78) > [junit] at > io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:53) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58) > [junit] at > io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47) > [junit] at > io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91) > [junit] at > io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68) > [junit] at > org.apache.cassandra.net.async.NettyFactory.getEventLoopGroup(NettyFactory.java:175) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Epoll(NettyFactoryTest.java:187) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_EpollWithoutIoRatioBoost(NettyFactoryTest.java:205) > [junit] Caused by: java.lang.ExceptionInInitializerError > [junit] at io.netty.channel.epoll.Epoll.(Epoll.java:33) > [junit] at > org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162) > [junit] at > org.apache.cassandra.net.async.NettyFactory.(NettyFactory.java:94) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Nio(NettyFactoryTest.java:216) > [junit] at > org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_NioWithoutIoRatioBoost(NettyFactoryTest.java:211) > [junit] Caused by: java.lang.IllegalStateException: Only supported on > Linux > [junit] at > io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189) > [junit] at io.netty.channel.epoll.Native.(Native.java:61) > {code} > It's obviously caused by epoll being unavailable on MacOS, but the tests > should still be made to pass. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153528#comment-16153528 ] Jason Brown commented on CASSANDRA-8457: [~spo...@gmail.com] ahh! you are correct. ninja-fixed as sha {{e38015ed1fcf7d95694683601b121c918c5f6976}} > nio MessagingService > > > Key: CASSANDRA-8457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8457 > Project: Cassandra > Issue Type: New Feature >Reporter: Jonathan Ellis >Assignee: Jason Brown >Priority: Minor > Labels: netty, performance > Fix For: 4.0 > > Attachments: 8457-load.tgz > > > Thread-per-peer (actually two each incoming and outbound) is a big > contributor to context switching, especially for larger clusters. Let's look > at switching to nio, possibly via Netty. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: ninja-fix build.xml netty version to 4.1.14 as jar is already updated in lib dir
Repository: cassandra Updated Branches: refs/heads/trunk 07c90c436 -> e38015ed1 ninja-fix build.xml netty version to 4.1.14 as jar is already updated in lib dir Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e38015ed Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e38015ed Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e38015ed Branch: refs/heads/trunk Commit: e38015ed1fcf7d95694683601b121c918c5f6976 Parents: 07c90c4 Author: Jason Brown Authored: Tue Sep 5 04:52:43 2017 -0700 Committer: Jason Brown Committed: Tue Sep 5 04:52:43 2017 -0700 -- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e38015ed/build.xml -- diff --git a/build.xml b/build.xml index ee22921..63518c5 100644 --- a/build.xml +++ b/build.xml @@ -420,7 +420,7 @@ - + - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153498#comment-16153498 ] Romain GERARD commented on CASSANDRA-13418: --- Champagne ! Thanks for the commitment :) > Allow TWCS to ignore overlaps when dropping fully expired sstables > -- > > Key: CASSANDRA-13418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13418 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Corentin Chary >Assignee: Romain GERARD > Labels: twcs > Fix For: 3.11.1, 4.0 > > Attachments: twcs-cleanup.png > > > http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If > you really want read-repairs you're going to have sstables blocking the > expiration of other fully expired SSTables because they overlap. > You can set unchecked_tombstone_compaction = true or tombstone_threshold to a > very low value and that will purge the blockers of old data that should > already have expired, thus removing the overlaps and allowing the other > SSTables to expire. > The thing is that this is rather CPU intensive and not optimal. If you have > time series, you might not care if all your data doesn't exactly expire at > the right time, or if data re-appears for some time, as long as it gets > deleted as soon as it can. And in this situation I believe it would be really > beneficial to allow users to simply ignore overlapping SSTables when looking > for fully expired ones. > To the question: why would you need read-repairs ? > - Full repairs basically take longer than the TTL of the data on my dataset, > so this isn't really effective. > - Even with a 10% chances of doing a repair, we found out that this would be > enough to greatly reduce entropy of the most used data (and if you have > timeseries, you're likely to have a dashboard doing the same important > queries over and over again). > - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow. > I'll try to come up with a patch demonstrating how this would work, try it on > our system and report the effects. > cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13842) dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test
[ https://issues.apache.org/jira/browse/CASSANDRA-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko reassigned CASSANDRA-13842: - Assignee: Aleksey Yeschenko > dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test > > > Key: CASSANDRA-13842 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13842 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Aleksey Yeschenko > > batch_test.TestBatch.batchlog_replay_compatibility_1_test and > batch_test.TestBatch.batchlog_replay_compatibility_4_test are failing: > http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_1_test/ > http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_4_test/ -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13842) dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test
[ https://issues.apache.org/jira/browse/CASSANDRA-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153492#comment-16153492 ] Aleksey Yeschenko commented on CASSANDRA-13842: --- The test passes 100% reliably on MacOS, but it does fail for me in my Ubuntu VM just as it does with Jenkins. Don't yet know why the difference is there, but I'll deal with it. > dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test > > > Key: CASSANDRA-13842 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13842 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson > > batch_test.TestBatch.batchlog_replay_compatibility_1_test and > batch_test.TestBatch.batchlog_replay_compatibility_4_test are failing: > http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_1_test/ > http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_4_test/ -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13808) Integer overflows with Amazon Elastic File System (EFS)
[ https://issues.apache.org/jira/browse/CASSANDRA-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Ishchenko updated CASSANDRA-13808: -- Description: Integer overflow issue was fixed for cassandra 2.2, but in 3.8 new property was introduced in config that also derives from disk size {{cdc_total_space_in_mb}}, see CASSANDRA-8844 It should be updated too https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484 was: Integer overflow issue was fixed for cassandra 2.2, but since then new property was introduced in config that also derives from disk size {{cdc_total_space_in_mb}} It should be updated too https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484 > Integer overflows with Amazon Elastic File System (EFS) > --- > > Key: CASSANDRA-13808 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13808 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Vitalii Ishchenko >Assignee: Benjamin Lerer > Fix For: 3.11.1 > > > Integer overflow issue was fixed for cassandra 2.2, but in 3.8 new property > was introduced in config that also derives from disk size > {{cdc_total_space_in_mb}}, see CASSANDRA-8844 > It should be updated too > https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13808) Integer overflows with Amazon Elastic File System (EFS)
[ https://issues.apache.org/jira/browse/CASSANDRA-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Ishchenko updated CASSANDRA-13808: -- Fix Version/s: 3.11.1 > Integer overflows with Amazon Elastic File System (EFS) > --- > > Key: CASSANDRA-13808 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13808 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Vitalii Ishchenko >Assignee: Benjamin Lerer > Fix For: 3.11.1 > > > Integer overflow issue was fixed for cassandra 2.2, but since then new > property was introduced in config that also derives from disk size > {{cdc_total_space_in_mb}} > It should be updated too > https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13263) Incorrect ComplexColumnData hashCode implementation
[ https://issues.apache.org/jira/browse/CASSANDRA-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13263: --- Resolution: Fixed Fix Version/s: 4.0 3.11.1 3.0.15 Status: Resolved (was: Patch Available) I'm pretty sure that fixing a hashCode function that isn't even used yet will cause much harm. Merged as 1e80e35 for 3.0 upwards with your suggested changes. Thanks for reviewing. > Incorrect ComplexColumnData hashCode implementation > --- > > Key: CASSANDRA-13263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13263 > Project: Cassandra > Issue Type: Bug >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski > Fix For: 3.0.15, 3.11.1, 4.0 > > > I went through some of the logs from CASSANDRA-13175 and one of the more > serious issues that we should address seem to be the > {{ComplexColumnData.hashCode()}} implementation. As Objects.hashCode is not > using deep hashing for array arguments, hashed will be based on the identity > instead of the array's content. See patch for simple fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[1/6] cassandra git commit: Fix ComplexColumnData hashCode implementation
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 fb92a5262 -> 1e80e3539 refs/heads/cassandra-3.11 5a09fafea -> 03155b021 refs/heads/trunk 9717cd67b -> 07c90c436 Fix ComplexColumnData hashCode implementation patch by Stefan Podkowinski; reviewed by Sylvain Lebresne for CASSANDRA-13263 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e80e353 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e80e353 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e80e353 Branch: refs/heads/cassandra-3.0 Commit: 1e80e35396e701bde97594bd49bb730656e02ce8 Parents: fb92a52 Author: Stefan Podkowinski Authored: Fri Feb 24 12:52:17 2017 +0100 Committer: Stefan Podkowinski Committed: Tue Sep 5 12:48:48 2017 +0200 -- src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e80e353/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java -- diff --git a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java index 2a8e843..e768769 100644 --- a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java +++ b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java @@ -218,7 +218,7 @@ public class ComplexColumnData extends ColumnData implements Iterable @Override public int hashCode() { -return Objects.hash(column(), complexDeletion(), cells); +return Objects.hash(column(), complexDeletion(), BTree.hashCode(cells)); } public static Builder builder() - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03155b02 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03155b02 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03155b02 Branch: refs/heads/trunk Commit: 03155b021a1232a0ef852ed836b68cde5175094f Parents: 5a09faf 1e80e35 Author: Stefan Podkowinski Authored: Tue Sep 5 12:51:08 2017 +0200 Committer: Stefan Podkowinski Committed: Tue Sep 5 12:52:17 2017 +0200 -- src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/03155b02/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03155b02 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03155b02 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03155b02 Branch: refs/heads/cassandra-3.11 Commit: 03155b021a1232a0ef852ed836b68cde5175094f Parents: 5a09faf 1e80e35 Author: Stefan Podkowinski Authored: Tue Sep 5 12:51:08 2017 +0200 Committer: Stefan Podkowinski Committed: Tue Sep 5 12:52:17 2017 +0200 -- src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/03155b02/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/07c90c43 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/07c90c43 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/07c90c43 Branch: refs/heads/trunk Commit: 07c90c4360c8a1558810013ddd1a26d10af15084 Parents: 9717cd6 03155b0 Author: Stefan Podkowinski Authored: Tue Sep 5 12:52:37 2017 +0200 Committer: Stefan Podkowinski Committed: Tue Sep 5 12:52:52 2017 +0200 -- src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/07c90c43/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[3/6] cassandra git commit: Fix ComplexColumnData hashCode implementation
Fix ComplexColumnData hashCode implementation patch by Stefan Podkowinski; reviewed by Sylvain Lebresne for CASSANDRA-13263 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e80e353 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e80e353 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e80e353 Branch: refs/heads/trunk Commit: 1e80e35396e701bde97594bd49bb730656e02ce8 Parents: fb92a52 Author: Stefan Podkowinski Authored: Fri Feb 24 12:52:17 2017 +0100 Committer: Stefan Podkowinski Committed: Tue Sep 5 12:48:48 2017 +0200 -- src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e80e353/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java -- diff --git a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java index 2a8e843..e768769 100644 --- a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java +++ b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java @@ -218,7 +218,7 @@ public class ComplexColumnData extends ColumnData implements Iterable @Override public int hashCode() { -return Objects.hash(column(), complexDeletion(), cells); +return Objects.hash(column(), complexDeletion(), BTree.hashCode(cells)); } public static Builder builder() - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[2/6] cassandra git commit: Fix ComplexColumnData hashCode implementation
Fix ComplexColumnData hashCode implementation patch by Stefan Podkowinski; reviewed by Sylvain Lebresne for CASSANDRA-13263 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e80e353 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e80e353 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e80e353 Branch: refs/heads/cassandra-3.11 Commit: 1e80e35396e701bde97594bd49bb730656e02ce8 Parents: fb92a52 Author: Stefan Podkowinski Authored: Fri Feb 24 12:52:17 2017 +0100 Committer: Stefan Podkowinski Committed: Tue Sep 5 12:48:48 2017 +0200 -- src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e80e353/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java -- diff --git a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java index 2a8e843..e768769 100644 --- a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java +++ b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java @@ -218,7 +218,7 @@ public class ComplexColumnData extends ColumnData implements Iterable @Override public int hashCode() { -return Objects.hash(column(), complexDeletion(), cells); +return Objects.hash(column(), complexDeletion(), BTree.hashCode(cells)); } public static Builder builder() - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13662) Remove unsupported CREDENTIALS message
[ https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153458#comment-16153458 ] Stefan Podkowinski commented on CASSANDRA-13662: I gave this another look and came up with a more reasonable test, but had to refactor Frame a bit in the process. Changes should be non-invasive enough for 4.0, wdyt? * [branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13662] * [testall|https://circleci.com/gh/spodkowinski/cassandra/119] > Remove unsupported CREDENTIALS message > -- > > Key: CASSANDRA-13662 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13662 > Project: Cassandra > Issue Type: Improvement > Components: Auth, CQL >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Minor > Labels: security > Fix For: 4.x > > > Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try > not to keep unused legacy classes around for any security relevant features. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13787) RangeTombstoneMarker and ParitionDeletion is not properly included in MV
[ https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhaoYang updated CASSANDRA-13787: - Description: Found two problems related to MV tombstone. 1. Range-tombstone-Marker being ignored after shadowing first row, subsequent base rows are not shadowed in TableViews. If the range tombstone was not flushed, it was used as deleted row to shadow new updates. It works correctly. After range tombstone was flushed, it was used as RangeTombstoneMarker and being skipped after shadowing first update. The bound of RangeTombstoneMarker seems wrong, it contained full clustering, but it should contain range or it should be multiple RangeTombstoneMarkers for multiple slices(aka. new updates) -2. Partition tombstone is not used when no existing live data, it will resurrect deleted cells. It was found in 11500 and included in that patch.- (Merged in CASSANDRA-11500) In order not to make 11500 patch more complicated, I will try fix range/partition tombstone issue here. {code:title=Tests to reproduce} @Test public void testExistingRangeTombstoneWithFlush() throws Throwable { testExistingRangeTombstone(true); } @Test public void testExistingRangeTombstoneWithoutFlush() throws Throwable { testExistingRangeTombstone(false); } public void testExistingRangeTombstone(boolean flush) throws Throwable { createTable("CREATE TABLE %s (k1 int, c1 int, c2 int, v1 int, v2 int, PRIMARY KEY (k1, c1, c2))"); execute("USE " + keyspace()); executeNet(protocolVersion, "USE " + keyspace()); createView("view1", "CREATE MATERIALIZED VIEW view1 AS SELECT * FROM %%s WHERE k1 IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL PRIMARY KEY (k1, c2, c1)"); updateView("DELETE FROM %s USING TIMESTAMP 10 WHERE k1 = 1 and c1=1"); if (flush) Keyspace.open(keyspace()).getColumnFamilyStore(currentTable()).forceBlockingFlush(); String table = KEYSPACE + "." + currentTable(); updateView("BEGIN BATCH " + "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 0, 0, 0) USING TIMESTAMP 5; " + "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 1, 0, 1) USING TIMESTAMP 5; " + "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 0, 1, 0) USING TIMESTAMP 5; " + "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 1, 1, 1) USING TIMESTAMP 5; " + "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 2, 1, 2) USING TIMESTAMP 5; " + "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 3, 1, 3) USING TIMESTAMP 5; " + "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 2, 0, 2, 0) USING TIMESTAMP 5; " + "APPLY BATCH"); assertRowsIgnoringOrder(execute("select * from %s"), row(1, 0, 0, 0, 0), row(1, 0, 1, 0, 1), row(1, 2, 0, 2, 0)); assertRowsIgnoringOrder(execute("select k1,c1,c2,v1,v2 from view1"), row(1, 0, 0, 0, 0), row(1, 0, 1, 0, 1), row(1, 2, 0, 2, 0)); } @Test public void testExistingParitionDeletionWithFlush() throws Throwable { testExistingParitionDeletion(true); } @Test public void testExistingParitionDeletionWithoutFlush() throws Throwable { testExistingParitionDeletion(false); } public void testExistingParitionDeletion(boolean flush) throws Throwable { // for partition range deletion, need to know that existing row is shadowed instead of not existed. createTable("CREATE TABLE %s (a int, b int, c int, d int, PRIMARY KEY (a))"); execute("USE " + keyspace()); executeNet(protocolVersion, "USE " + keyspace()); createView("mv_test1", "CREATE MATERIALIZED VIEW %s AS SELECT * FROM %%s WHERE a IS NOT NULL AND b IS NOT NULL PRIMARY KEY (a, b)"); Keyspace ks = Keyspace.open(keyspace()); ks.getColumnFamilyStore("mv_test1").disableAutoCompaction(); execute("INSERT INTO %s (a, b, c, d) VALUES (?, ?, ?, ?) using timestamp 0", 1, 1, 1, 1); if (flush) FBUtilities.waitOnFutures(ks.flush()); assertRowsIgnoringOrder(execute("SELECT * FROM mv_test1"), row(1, 1, 1, 1)); // remove view row updateView("UPDATE %s using timestamp 1 set b = null WHERE a=1"); if (flush) FBUtilities.waitOnFutures(ks.flush()); assertRowsIgnoringOrder(execute("SELECT * FROM mv_test1")); // remove base row, no view updated generated. updateView("DELETE FROM %s using timestamp 2 wh
[jira] [Commented] (CASSANDRA-12373) 3.0 breaks CQL compatibility with super columns families
[ https://issues.apache.org/jira/browse/CASSANDRA-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153401#comment-16153401 ] Sylvain Lebresne commented on CASSANDRA-12373: -- Thanks for the update and great job on the much needed additional testing. I think we're getting there but I still have a bunch of remarks :) * In {{CFMetaData}} ** In the ctor, for {{isDense}}, we seem to force recomputation for non-SC non-dense tables (which happens to include non-compact tables). I don't think thats what we want (and the {{recalculateIsDense}} is obviously meant only for SC). But for I'm also not 100% sure this work in all upgrade scenario. Typically, if a dense SCF is upgraded to a current 3.x version, I believe it will end up with 2 regular column definition: the first coming from the fact 2.x will have had a "COMPACT_VALUE" column definition, which will become a REGULAR when upgraded, and second being the "super column map" column that is force-added to all SCF. And so that re-computation code will not detect things properly. Note that I haven't actually tested this tbh so I could be wrong, but this highlight that we really need to be running some SC upgrade dtest both for {{2.x \-> 3.x+}} and {{2.x \-> current 3.x \-> 3.x+}} because things are imo too complex to rely on reasoning alone. Unfortunately, while we were supposed to have a [SC upgrade dtests|https://github.com/apache/cassandra-dtest/blob/master/upgrade_tests/upgrade_supercolumns_test.py], it appears [this commit|https://github.com/apache/cassandra-dtest/commit/7f47d4d82d6becdb9fd267bd3844d32aee6acdea] has made this test useless by making it (confusingly) not about super column families. So we'll need to revert that commit somehow and the improve the test to include both dense and sparse SC. ** In {{rebuild()}}, why not move the SC code after {{columnMetadata}} has been populated so it can be simplified? Nothing seems to rely on it being done _before_ {{columnMetadata}} is populated. * I'm not entirely sure if/how column renaming works for the SC key and value "fake" column (would have assumed {{AlterTableStatement}} would need a least a bit of special casing to recognize those "fake" column). It would be nice to add at test to check that renames still work like in 2.2 for those columns. * In {{UpdateParameters#addCounter}}, not sure the change from {{CounterContext#createUpdate}} to {{CounterContext#createLocal}} is corret. Probably a bad post-CASSANDRA-13691 rebase? * In {{SuperColumnCompatibility}: ** The first line of the javadoc states that it's only for "dense" SC, but it's not 100% true as things like {{SUPER_COLUMN_MAP_COLUMN}} are used for all SC. And the rest of the javadoc describing the internal layout also applies to all SC. Bit of a detail I suppose, but I'd rephrase things to avoid any confusion. ** In {{getSuperCfKeyColumn}}, not sure what "3.x-created supercolumn family" refers to (is that SC tables created from thrift in 3.x? or SC tables whose schema had been upgraded by a previous 3.x version?) and more importantly, why it would differ in number of clustering columns? ** I'm confused by the code of {{getSuperCfValueColumn}}: dense tables are supposed to have only a single regular column, and {{cfm.compactValueColumn()}} is supposed to alias it. So I'm not sure what the loop is trying to do, except basically returning {{cfm.compactValueColumn()}}, nor when that loop wouldn't return (all SC should have the "super column map" internally. Further, due to what's above, said loop appears to return a definition whose type is the {{MapType}}, while the code following it clearly return a definition whose type is the values of said {{MapType}}, so it feels wrong the method would return different kind of types depending on the path taken. I'm most likely missing something since your tests seem to be working, but I'm not sure what. ** In {{processPartition}}: *** There is a typo in a comment, where in "... can't be change to support inclusive slice ...", I believe you want "exclusive". *** In the {{REGULAR}} branch of the {{switch}}: in the 2nd branch ({{else if ...}}), we should use {{result.add(cell)}} directly. Not only does that already handle counters, but more importantly, if we don't do that, things like {{ttl(v)}} will not work on the compact column value. the last {{else}} is confusing: by definition we shouldn't be exposing any other column than the "compact value" column, so that branch should probably not be there (or be an assertion that we shouldn't get there). ** In the {{updates}} loop in {{prepareUpdateOperations}}, not sure why the 2 first cases seem to assume that {{cfm.isSuperColumnValueColumn(def)}}, but the 3rd case tests it explicitely (but after having assigned {{superColumnValue}}). In general, not sure why the 3rd case looks different from the other 2.
[jira] [Commented] (CASSANDRA-13809) Make BatchlogManagerMBean.forceBatchlogReplay() blocking
[ https://issues.apache.org/jira/browse/CASSANDRA-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153365#comment-16153365 ] Marcus Eriksson commented on CASSANDRA-13809: - created CASSANDRA-13842 > Make BatchlogManagerMBean.forceBatchlogReplay() blocking > > > Key: CASSANDRA-13809 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13809 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 2.2.11 > > > In 3.0, {{BatchlogManagerMBean.forceBatchlogReplay()}} blocks until > completion. In 2.2 it just submits a runnable and instaexits, which makes it > impossible to create non-flaky dtests that rely on batchlog replay. > [here|https://github.com/iamaleksey/cassandra/commits/13809-2.2] is a small > 2.2-only commit that make the behaviour consistent between 2.2 and 3.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13842) dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test
Marcus Eriksson created CASSANDRA-13842: --- Summary: dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test Key: CASSANDRA-13842 URL: https://issues.apache.org/jira/browse/CASSANDRA-13842 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson batch_test.TestBatch.batchlog_replay_compatibility_1_test and batch_test.TestBatch.batchlog_replay_compatibility_4_test are failing: http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_1_test/ http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_4_test/ -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13809) Make BatchlogManagerMBean.forceBatchlogReplay() blocking
[ https://issues.apache.org/jira/browse/CASSANDRA-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153360#comment-16153360 ] Aleksey Yeschenko commented on CASSANDRA-13809: --- The dtests aren't for it, it's the other way around. I have no idea how and why are they failing, however. Ran each of them dozens of times locally after I started seeing the failures on jenkins, and they never fail. > Make BatchlogManagerMBean.forceBatchlogReplay() blocking > > > Key: CASSANDRA-13809 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13809 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko >Priority: Trivial > Fix For: 2.2.11 > > > In 3.0, {{BatchlogManagerMBean.forceBatchlogReplay()}} blocks until > completion. In 2.2 it just submits a runnable and instaexits, which makes it > impossible to create non-flaky dtests that rely on batchlog replay. > [here|https://github.com/iamaleksey/cassandra/commits/13809-2.2] is a small > 2.2-only commit that make the behaviour consistent between 2.2 and 3.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[ https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153337#comment-16153337 ] ZhaoYang commented on CASSANDRA-11500: -- [~pauloricardomg] (y) thanks for the feedback, review, improvements. > Obsolete MV entry may not be properly deleted > - > > Key: CASSANDRA-11500 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11500 > Project: Cassandra > Issue Type: Bug > Components: Materialized Views >Reporter: Sylvain Lebresne >Assignee: ZhaoYang > Fix For: 3.0.15, 3.11.1, 4.0 > > > When a Materialized View uses a non-PK base table column in its PK, if an > update changes that column value, we add the new view entry and remove the > old one. When doing that removal, the current code uses the same timestamp > than for the liveness info of the new entry, which is the max timestamp for > any columns participating to the view PK. This is not correct for the > deletion as the old view entry could have other columns with higher timestamp > which won't be deleted as can easily shown by the failing of the following > test: > {noformat} > CREATE TABLE t (k int PRIMARY KEY, a int, b int); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (k, a); > INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0; > UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1; > UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; > SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old > (invalid) and the new one > {noformat} > So the correct timestamp to use for the deletion is the biggest timestamp in > the old view entry (which we know since we read the pre-existing base row), > and that is what CASSANDRA-11475 does (the test above thus doesn't fail on > that branch). > Unfortunately, even then we can still have problems if further updates > requires us to overide the old entry. Consider the following case: > {noformat} > CREATE TABLE t (k int PRIMARY KEY, a int, b int); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (k, a); > INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0; > UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1; > UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the > entry for a=1 with timestamp 10 > UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert > an entry for a=1 but shouldn't be deleted by the prior deletion > UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this > game more than once > UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1; > ... > {noformat} > In a way, this is saying that the "shadowable" deletion mechanism is not > general enough: we need to be able to re-insert an entry when a prior one had > been deleted before, but we can't rely on timestamps being strictly bigger on > the re-insert. In that sense, this can be though as a similar problem than > CASSANDRA-10965, though the solution there of a single flag is not enough > since we can have to replace more than once. > I think the proper solution would be to ship enough information to always be > able to decide when a view deletion is shadowed. Which means that both > liveness info (for updates) and shadowable deletion would need to ship the > timestamp of any base table column that is part the view PK (so {{a}} in the > example below). It's doable (and not that hard really), but it does require > a change to the sstable and intra-node protocol, which makes this a bit > painful right now. > But I'll also note that as CASSANDRA-1096 shows, the timestamp is not even > enough since on equal timestamp the value can be the deciding factor. So in > theory we'd have to ship the value of those columns (in the case of a > deletion at least since we have it in the view PK for updates). That said, on > that last problem, my preference would be that we start prioritizing > CASSANDRA-6123 seriously so we don't have to care about conflicting timestamp > anymore, which would make this problem go away. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[ https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-13738: Resolution: Fixed Fix Version/s: (was: 3.11.x) (was: 4.x) (was: 3.0.x) (was: 2.2.x) 4.0 3.11.1 3.0.15 2.2.11 Status: Resolved (was: Patch Available) > Load is over calculated after each IndexSummaryRedistribution > - > > Key: CASSANDRA-13738 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13738 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jay Zhuang >Assignee: Jay Zhuang > Fix For: 2.2.11, 3.0.15, 3.11.1, 4.0 > > Attachments: sizeIssue.png > > > For example, here is one of our cluster with about 500GB per node, but > {{nodetool status}} shows far more load than it actually is and keeps > increasing, restarting the process will reset the load, but keeps increasing > afterwards: > {noformat} > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- AddressLoad Tokens Owns (effective) Host ID > Rack > UN IP1* 13.52 TB 256 100.0% > c4c31e0a-3f01-49f7-8a22-33043737975d rac1 > UN IP2* 14.25 TB 256 100.0% > efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1 > UN IP3* 13.52 TB 256 100.0% > 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1 > UN IP4* 22.13 TB 256 100.0% > 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1 > UN IP5* 18.02 TB 256 100.0% > 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1 > UN IP6* 11.68 TB 256 100.0% > d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1 > {noformat} > !sizeIssue.png|test! > The root cause is if the SSTable index summary is redistributed (typically > executes hourly), the updated SSTable size is added again. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution
[ https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153283#comment-16153283 ] Marcus Eriksson commented on CASSANDRA-13738: - failing dtests pass locally committed as {{4e834c53ca57910e8c4}}, thanks! > Load is over calculated after each IndexSummaryRedistribution > - > > Key: CASSANDRA-13738 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13738 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jay Zhuang >Assignee: Jay Zhuang > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x > > Attachments: sizeIssue.png > > > For example, here is one of our cluster with about 500GB per node, but > {{nodetool status}} shows far more load than it actually is and keeps > increasing, restarting the process will reset the load, but keeps increasing > afterwards: > {noformat} > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- AddressLoad Tokens Owns (effective) Host ID > Rack > UN IP1* 13.52 TB 256 100.0% > c4c31e0a-3f01-49f7-8a22-33043737975d rac1 > UN IP2* 14.25 TB 256 100.0% > efec4980-ec9e-4424-8a21-ce7ddaf80aa0 rac1 > UN IP3* 13.52 TB 256 100.0% > 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8 rac1 > UN IP4* 22.13 TB 256 100.0% > 8879e6c4-93e3-4cc5-b957-f999c6b9b563 rac1 > UN IP5* 18.02 TB 256 100.0% > 4a1eaf22-4a83-4736-9e1c-12f898d685fa rac1 > UN IP6* 11.68 TB 256 100.0% > d633c591-28af-42cc-bc5e-47d1c8bcf50f rac1 > {noformat} > !sizeIssue.png|test! > The root cause is if the SSTable index summary is redistributed (typically > executes hourly), the updated SSTable size is added again. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[03/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution
Fix load over calculated issue in IndexSummaryRedistribution patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53 Branch: refs/heads/cassandra-3.11 Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556 Parents: e80ede6 Author: Jay Zhuang Authored: Wed Aug 2 21:44:39 2017 -0700 Committer: Marcus Eriksson Committed: Tue Sep 5 10:33:39 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 6 +- .../sstable/IndexSummaryRedistributionTest.java | 145 +++ 3 files changed, 149 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 03a78fd..1abd7de 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.11 + * Fix load over calculated issue in IndexSummaryRedistribution (CASSANDRA-13738) * Fix compaction and flush exception not captured (CASSANDRA-13833) * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809) * Uncaught exceptions in Netty pipeline (CASSANDRA-13649) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index ba060d4..885 100644 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java -- diff --git a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java new file mode 100644 index 000..77fd69a --- /dev/null +++ b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.io.sstable; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; + +import org.junit.BeforeClass; +import org.junit.Test; + +import org.apache.cassandra.SchemaLoader; +import org.apache.cassandra.Util; +import org.apache.cassandra.cache.CachingOptions; +import org.apache.cassandra.config.KSMetaData; +import org.apache.cassandra.db.ColumnFamilyStore; +import org.apache.cassandra.db.DecoratedKey; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.db.Mutation; +import org.apache.cassandra.exceptions.ConfigurationException; +import org.apache.cassandra.io.sstable.format.SSTableReader; +import org.apache.cassandra.locator.SimpleStrategy; +import org.apache.cassandra.metrics.RestorableMeter; +import org.apache.cassandra.metrics.StorageMetrics; + +import static org.junit.Assert.assertEquals; + +public class IndexSummaryRedistributionTest +{ +private static final String KEYSPACE1 = "IndexSummaryRedistributionTest"; +private static final String CF_STANDARD = "Standard"; + +@BeforeClass +public static void defineSchema() throws ConfigurationException +{ +SchemaLoader.prepareServer(); +SchemaLoader.createKeyspace(KEYSPACE1, +SimpleSt
[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5a09fafe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5a09fafe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5a09fafe Branch: refs/heads/cassandra-3.11 Commit: 5a09fafea043d472f2d46b1e17062201f7baa38a Parents: 2b29f8a fb92a52 Author: Marcus Eriksson Authored: Tue Sep 5 10:39:02 2017 +0200 Committer: Marcus Eriksson Committed: Tue Sep 5 10:39:02 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 8 +- .../sstable/IndexSummaryRedistributionTest.java | 142 +++ 3 files changed, 147 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index 03af2bb,25ba212..5d76f75 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@@ -1143,13 -1116,17 +1143,13 @@@ public abstract class SSTableReader ext "no adjustments to min/max_index_interval"); } -//Always save the resampled index -try(SegmentedFile.Builder ibuilder = SegmentedFile.getBuilder(DatabaseDescriptor.getIndexAccessMode(), false); -SegmentedFile.Builder dbuilder = SegmentedFile.getBuilder(DatabaseDescriptor.getDiskAccessMode(), compression)) -{ -saveSummary(ibuilder, dbuilder, newSummary); -} +// Always save the resampled index +saveSummary(newSummary); - long newSize = bytesOnDisk(); - StorageMetrics.load.inc(newSize - oldSize); - parent.metric.liveDiskSpaceUsed.inc(newSize - oldSize); - parent.metric.totalDiskSpaceUsed.inc(newSize - oldSize); + // The new size will be added in Transactional.commit() as an updated SSTable, more details: CASSANDRA-13738 + StorageMetrics.load.dec(oldSize); + parent.metric.liveDiskSpaceUsed.dec(oldSize); + parent.metric.totalDiskSpaceUsed.dec(oldSize); return cloneAndReplace(first, OpenReason.METADATA_CHANGE, newSummary); } - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fb92a526 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fb92a526 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fb92a526 Branch: refs/heads/trunk Commit: fb92a52624c13969386760b967b7168a87e857bb Parents: f4014bb 4e834c5 Author: Marcus Eriksson Authored: Tue Sep 5 10:38:39 2017 +0200 Committer: Marcus Eriksson Committed: Tue Sep 5 10:38:39 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 8 +- .../sstable/IndexSummaryRedistributionTest.java | 142 +++ 3 files changed, 147 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/CHANGES.txt -- diff --cc CHANGES.txt index 870351c,1abd7de..1ed3fe6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,39 -1,9 +1,40 @@@ -2.2.11 +3.0.15 + * Fix MV timestamp issues (CASSANDRA-11500) + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) + * Fix race condition in read command serialization (CASSANDRA-13363) + * Enable segement creation before recovering commitlogs (CASSANDRA-13587) + * Fix AssertionError in short read protection (CASSANDRA-13747) + * Don't skip corrupted sstables on startup (CASSANDRA-13620) + * Fix the merging of cells with different user type versions (CASSANDRA-13776) + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640) + * Potential AssertionError during ReadRepair of range tombstone and partition deletions (CASSANDRA-13719) + * Don't let stress write warmup data if n=0 (CASSANDRA-13773) + * Gossip thread slows down when using batch commit log (CASSANDRA-12966) + * Randomize batchlog endpoint selection with only 1 or 2 racks (CASSANDRA-12884) + * Fix digest calculation for counter cells (CASSANDRA-13750) + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change SSTabledump to use type.toJSONString() (CASSANDRA-13573) + * Skip materialized view addition if the base table doesn't exist (CASSANDRA-13737) + * Drop table should remove corresponding entries in dropped_columns table (CASSANDRA-13730) + * Log warn message until legacy auth tables have been migrated (CASSANDRA-13371) + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 (CASSANDRA-13691) + * Fix invalid writetime for null cells (CASSANDRA-13711) + * Fix ALTER TABLE statement to atomically propagate changes to the table and its MVs (CASSANDRA-12952) + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722) + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize (Backport CASSANDRA-13329) + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily (CASSANDRA-13696) + * Purge tombstones created by expired cells (CASSANDRA-13643) + * Make concat work with iterators that have different subsets of columns (CASSANDRA-13482) + * Set test.runners based on cores and memory size (CASSANDRA-13078) + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557) + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606) + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627) + * Nodetool listsnapshots output is missing a newline, if there are no snapshots (CASSANDRA-13568) + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532) +Merged from 2.2: + * Fix load over calculated issue in IndexSummaryRedistribution (CASSANDRA-13738) * Fix compaction and flush exception not captured (CASSANDRA-13833) - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809) * Uncaught exceptions in Netty pipeline (CASSANDRA-13649) - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223) * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272) * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592) http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index d56b3e7,885..25ba212 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@@ -1123,10 -1169,14 +1123,10 @@@ public abstract class SSTableReader ext saveSummary(ibuilder, db
[01/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 e80ede6d3 -> 4e834c53c refs/heads/cassandra-3.0 f4014bb0f -> fb92a5262 refs/heads/cassandra-3.11 2b29f8a78 -> 5a09fafea refs/heads/trunk 65f7216c6 -> 9717cd67b Fix load over calculated issue in IndexSummaryRedistribution patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53 Branch: refs/heads/cassandra-2.2 Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556 Parents: e80ede6 Author: Jay Zhuang Authored: Wed Aug 2 21:44:39 2017 -0700 Committer: Marcus Eriksson Committed: Tue Sep 5 10:33:39 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 6 +- .../sstable/IndexSummaryRedistributionTest.java | 145 +++ 3 files changed, 149 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 03a78fd..1abd7de 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.11 + * Fix load over calculated issue in IndexSummaryRedistribution (CASSANDRA-13738) * Fix compaction and flush exception not captured (CASSANDRA-13833) * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809) * Uncaught exceptions in Netty pipeline (CASSANDRA-13649) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index ba060d4..885 100644 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java -- diff --git a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java new file mode 100644 index 000..77fd69a --- /dev/null +++ b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.io.sstable; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; + +import org.junit.BeforeClass; +import org.junit.Test; + +import org.apache.cassandra.SchemaLoader; +import org.apache.cassandra.Util; +import org.apache.cassandra.cache.CachingOptions; +import org.apache.cassandra.config.KSMetaData; +import org.apache.cassandra.db.ColumnFamilyStore; +import org.apache.cassandra.db.DecoratedKey; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.db.Mutation; +import org.apache.cassandra.exceptions.ConfigurationException; +import org.apache.cassandra.io.sstable.format.SSTableReader; +import org.apache.cassandra.locator.SimpleStrategy; +import org.apache.cassandra.metrics.RestorableMeter; +import org.apache.cassandra.metrics.StorageMetrics; + +import static org.junit.Assert.assertEquals; + +public class IndexSummaryRedistributionTest +{ +private static final String KEYSPACE1 = "IndexSummaryRedistributionTest"; +private static final String CF_STANDARD = "Stand
[04/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution
Fix load over calculated issue in IndexSummaryRedistribution patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53 Branch: refs/heads/trunk Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556 Parents: e80ede6 Author: Jay Zhuang Authored: Wed Aug 2 21:44:39 2017 -0700 Committer: Marcus Eriksson Committed: Tue Sep 5 10:33:39 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 6 +- .../sstable/IndexSummaryRedistributionTest.java | 145 +++ 3 files changed, 149 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 03a78fd..1abd7de 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.11 + * Fix load over calculated issue in IndexSummaryRedistribution (CASSANDRA-13738) * Fix compaction and flush exception not captured (CASSANDRA-13833) * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809) * Uncaught exceptions in Netty pipeline (CASSANDRA-13649) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index ba060d4..885 100644 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java -- diff --git a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java new file mode 100644 index 000..77fd69a --- /dev/null +++ b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.io.sstable; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; + +import org.junit.BeforeClass; +import org.junit.Test; + +import org.apache.cassandra.SchemaLoader; +import org.apache.cassandra.Util; +import org.apache.cassandra.cache.CachingOptions; +import org.apache.cassandra.config.KSMetaData; +import org.apache.cassandra.db.ColumnFamilyStore; +import org.apache.cassandra.db.DecoratedKey; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.db.Mutation; +import org.apache.cassandra.exceptions.ConfigurationException; +import org.apache.cassandra.io.sstable.format.SSTableReader; +import org.apache.cassandra.locator.SimpleStrategy; +import org.apache.cassandra.metrics.RestorableMeter; +import org.apache.cassandra.metrics.StorageMetrics; + +import static org.junit.Assert.assertEquals; + +public class IndexSummaryRedistributionTest +{ +private static final String KEYSPACE1 = "IndexSummaryRedistributionTest"; +private static final String CF_STANDARD = "Standard"; + +@BeforeClass +public static void defineSchema() throws ConfigurationException +{ +SchemaLoader.prepareServer(); +SchemaLoader.createKeyspace(KEYSPACE1, +SimpleStrategy.cl
[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fb92a526 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fb92a526 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fb92a526 Branch: refs/heads/cassandra-3.0 Commit: fb92a52624c13969386760b967b7168a87e857bb Parents: f4014bb 4e834c5 Author: Marcus Eriksson Authored: Tue Sep 5 10:38:39 2017 +0200 Committer: Marcus Eriksson Committed: Tue Sep 5 10:38:39 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 8 +- .../sstable/IndexSummaryRedistributionTest.java | 142 +++ 3 files changed, 147 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/CHANGES.txt -- diff --cc CHANGES.txt index 870351c,1abd7de..1ed3fe6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,39 -1,9 +1,40 @@@ -2.2.11 +3.0.15 + * Fix MV timestamp issues (CASSANDRA-11500) + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) + * Fix race condition in read command serialization (CASSANDRA-13363) + * Enable segement creation before recovering commitlogs (CASSANDRA-13587) + * Fix AssertionError in short read protection (CASSANDRA-13747) + * Don't skip corrupted sstables on startup (CASSANDRA-13620) + * Fix the merging of cells with different user type versions (CASSANDRA-13776) + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640) + * Potential AssertionError during ReadRepair of range tombstone and partition deletions (CASSANDRA-13719) + * Don't let stress write warmup data if n=0 (CASSANDRA-13773) + * Gossip thread slows down when using batch commit log (CASSANDRA-12966) + * Randomize batchlog endpoint selection with only 1 or 2 racks (CASSANDRA-12884) + * Fix digest calculation for counter cells (CASSANDRA-13750) + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change SSTabledump to use type.toJSONString() (CASSANDRA-13573) + * Skip materialized view addition if the base table doesn't exist (CASSANDRA-13737) + * Drop table should remove corresponding entries in dropped_columns table (CASSANDRA-13730) + * Log warn message until legacy auth tables have been migrated (CASSANDRA-13371) + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 (CASSANDRA-13691) + * Fix invalid writetime for null cells (CASSANDRA-13711) + * Fix ALTER TABLE statement to atomically propagate changes to the table and its MVs (CASSANDRA-12952) + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722) + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize (Backport CASSANDRA-13329) + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily (CASSANDRA-13696) + * Purge tombstones created by expired cells (CASSANDRA-13643) + * Make concat work with iterators that have different subsets of columns (CASSANDRA-13482) + * Set test.runners based on cores and memory size (CASSANDRA-13078) + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557) + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606) + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627) + * Nodetool listsnapshots output is missing a newline, if there are no snapshots (CASSANDRA-13568) + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532) +Merged from 2.2: + * Fix load over calculated issue in IndexSummaryRedistribution (CASSANDRA-13738) * Fix compaction and flush exception not captured (CASSANDRA-13833) - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809) * Uncaught exceptions in Netty pipeline (CASSANDRA-13649) - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223) * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272) * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592) http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index d56b3e7,885..25ba212 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@@ -1123,10 -1169,14 +1123,10 @@@ public abstract class SSTableReader ext saveSummary(ibui
[02/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution
Fix load over calculated issue in IndexSummaryRedistribution patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53 Branch: refs/heads/cassandra-3.0 Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556 Parents: e80ede6 Author: Jay Zhuang Authored: Wed Aug 2 21:44:39 2017 -0700 Committer: Marcus Eriksson Committed: Tue Sep 5 10:33:39 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 6 +- .../sstable/IndexSummaryRedistributionTest.java | 145 +++ 3 files changed, 149 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 03a78fd..1abd7de 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.11 + * Fix load over calculated issue in IndexSummaryRedistribution (CASSANDRA-13738) * Fix compaction and flush exception not captured (CASSANDRA-13833) * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809) * Uncaught exceptions in Netty pipeline (CASSANDRA-13649) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index ba060d4..885 100644 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java -- diff --git a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java new file mode 100644 index 000..77fd69a --- /dev/null +++ b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java @@ -0,0 +1,145 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.cassandra.io.sstable; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Future; + +import org.junit.BeforeClass; +import org.junit.Test; + +import org.apache.cassandra.SchemaLoader; +import org.apache.cassandra.Util; +import org.apache.cassandra.cache.CachingOptions; +import org.apache.cassandra.config.KSMetaData; +import org.apache.cassandra.db.ColumnFamilyStore; +import org.apache.cassandra.db.DecoratedKey; +import org.apache.cassandra.db.Keyspace; +import org.apache.cassandra.db.Mutation; +import org.apache.cassandra.exceptions.ConfigurationException; +import org.apache.cassandra.io.sstable.format.SSTableReader; +import org.apache.cassandra.locator.SimpleStrategy; +import org.apache.cassandra.metrics.RestorableMeter; +import org.apache.cassandra.metrics.StorageMetrics; + +import static org.junit.Assert.assertEquals; + +public class IndexSummaryRedistributionTest +{ +private static final String KEYSPACE1 = "IndexSummaryRedistributionTest"; +private static final String CF_STANDARD = "Standard"; + +@BeforeClass +public static void defineSchema() throws ConfigurationException +{ +SchemaLoader.prepareServer(); +SchemaLoader.createKeyspace(KEYSPACE1, +SimpleStr
[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fb92a526 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fb92a526 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fb92a526 Branch: refs/heads/cassandra-3.11 Commit: fb92a52624c13969386760b967b7168a87e857bb Parents: f4014bb 4e834c5 Author: Marcus Eriksson Authored: Tue Sep 5 10:38:39 2017 +0200 Committer: Marcus Eriksson Committed: Tue Sep 5 10:38:39 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 8 +- .../sstable/IndexSummaryRedistributionTest.java | 142 +++ 3 files changed, 147 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/CHANGES.txt -- diff --cc CHANGES.txt index 870351c,1abd7de..1ed3fe6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,39 -1,9 +1,40 @@@ -2.2.11 +3.0.15 + * Fix MV timestamp issues (CASSANDRA-11500) + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) + * Fix race condition in read command serialization (CASSANDRA-13363) + * Enable segement creation before recovering commitlogs (CASSANDRA-13587) + * Fix AssertionError in short read protection (CASSANDRA-13747) + * Don't skip corrupted sstables on startup (CASSANDRA-13620) + * Fix the merging of cells with different user type versions (CASSANDRA-13776) + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640) + * Potential AssertionError during ReadRepair of range tombstone and partition deletions (CASSANDRA-13719) + * Don't let stress write warmup data if n=0 (CASSANDRA-13773) + * Gossip thread slows down when using batch commit log (CASSANDRA-12966) + * Randomize batchlog endpoint selection with only 1 or 2 racks (CASSANDRA-12884) + * Fix digest calculation for counter cells (CASSANDRA-13750) + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change SSTabledump to use type.toJSONString() (CASSANDRA-13573) + * Skip materialized view addition if the base table doesn't exist (CASSANDRA-13737) + * Drop table should remove corresponding entries in dropped_columns table (CASSANDRA-13730) + * Log warn message until legacy auth tables have been migrated (CASSANDRA-13371) + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 (CASSANDRA-13691) + * Fix invalid writetime for null cells (CASSANDRA-13711) + * Fix ALTER TABLE statement to atomically propagate changes to the table and its MVs (CASSANDRA-12952) + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722) + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize (Backport CASSANDRA-13329) + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily (CASSANDRA-13696) + * Purge tombstones created by expired cells (CASSANDRA-13643) + * Make concat work with iterators that have different subsets of columns (CASSANDRA-13482) + * Set test.runners based on cores and memory size (CASSANDRA-13078) + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557) + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606) + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627) + * Nodetool listsnapshots output is missing a newline, if there are no snapshots (CASSANDRA-13568) + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532) +Merged from 2.2: + * Fix load over calculated issue in IndexSummaryRedistribution (CASSANDRA-13738) * Fix compaction and flush exception not captured (CASSANDRA-13833) - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809) * Uncaught exceptions in Netty pipeline (CASSANDRA-13649) - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) * Fix queries with LIMIT and filtering on clustering columns (CASSANDRA-11223) * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272) * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592) http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index d56b3e7,885..25ba212 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@@ -1123,10 -1169,14 +1123,10 @@@ public abstract class SSTableReader ext saveSummary(ibu
[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9717cd67 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9717cd67 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9717cd67 Branch: refs/heads/trunk Commit: 9717cd67bd327f527bd97775057d356b674f50ec Parents: 65f7216 5a09faf Author: Marcus Eriksson Authored: Tue Sep 5 10:41:24 2017 +0200 Committer: Marcus Eriksson Committed: Tue Sep 5 10:41:50 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 8 +- .../sstable/IndexSummaryRedistributionTest.java | 143 +++ 3 files changed, 148 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9717cd67/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9717cd67/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9717cd67/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java -- diff --cc test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java index 000,31a57e1..07a2212 mode 00,100644..100644 --- a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java +++ b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java @@@ -1,0 -1,142 +1,143 @@@ + /* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + package org.apache.cassandra.io.sstable; + + import java.io.IOException; + import java.nio.ByteBuffer; + import java.util.ArrayList; + import java.util.List; + import java.util.concurrent.ExecutionException; + import java.util.concurrent.Future; + + import org.junit.BeforeClass; + import org.junit.Test; + + import org.apache.cassandra.SchemaLoader; + import org.apache.cassandra.db.ColumnFamilyStore; + import org.apache.cassandra.db.Keyspace; + import org.apache.cassandra.db.RowUpdateBuilder; + import org.apache.cassandra.exceptions.ConfigurationException; + import org.apache.cassandra.io.sstable.format.SSTableReader; + import org.apache.cassandra.metrics.RestorableMeter; + import org.apache.cassandra.metrics.StorageMetrics; + import org.apache.cassandra.schema.CachingParams; + import org.apache.cassandra.schema.KeyspaceParams; ++import org.apache.cassandra.schema.MigrationManager; + + import static org.junit.Assert.assertEquals; + + public class IndexSummaryRedistributionTest + { + private static final String KEYSPACE1 = "IndexSummaryRedistributionTest"; + private static final String CF_STANDARD = "Standard"; + + @BeforeClass + public static void defineSchema() throws ConfigurationException + { + SchemaLoader.prepareServer(); + SchemaLoader.createKeyspace(KEYSPACE1, + KeyspaceParams.simple(1), + SchemaLoader.standardCFMD(KEYSPACE1, CF_STANDARD) + .minIndexInterval(8) + .maxIndexInterval(256) + .caching(CachingParams.CACHE_NOTHING)); + } + + @Test + public void testMetricsLoadAfterRedistribution() throws IOException + { + String ksname = KEYSPACE1; + String cfname = CF_STANDARD; + Keyspace keyspace = Keyspace.open(ksname); + ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cfname); + int numSSTables = 1; + int numRows = 1024 * 10; + long load = StorageMetrics.load.getCount(); + StorageMetrics.load.dec(load); // reset the load metric + createSSTables(ksname, cfname, numSSTables, numRows); + + List sstables = new Arra
[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5a09fafe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5a09fafe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5a09fafe Branch: refs/heads/trunk Commit: 5a09fafea043d472f2d46b1e17062201f7baa38a Parents: 2b29f8a fb92a52 Author: Marcus Eriksson Authored: Tue Sep 5 10:39:02 2017 +0200 Committer: Marcus Eriksson Committed: Tue Sep 5 10:39:02 2017 +0200 -- CHANGES.txt | 1 + .../io/sstable/format/SSTableReader.java| 8 +- .../sstable/IndexSummaryRedistributionTest.java | 142 +++ 3 files changed, 147 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index 03af2bb,25ba212..5d76f75 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@@ -1143,13 -1116,17 +1143,13 @@@ public abstract class SSTableReader ext "no adjustments to min/max_index_interval"); } -//Always save the resampled index -try(SegmentedFile.Builder ibuilder = SegmentedFile.getBuilder(DatabaseDescriptor.getIndexAccessMode(), false); -SegmentedFile.Builder dbuilder = SegmentedFile.getBuilder(DatabaseDescriptor.getDiskAccessMode(), compression)) -{ -saveSummary(ibuilder, dbuilder, newSummary); -} +// Always save the resampled index +saveSummary(newSummary); - long newSize = bytesOnDisk(); - StorageMetrics.load.inc(newSize - oldSize); - parent.metric.liveDiskSpaceUsed.inc(newSize - oldSize); - parent.metric.totalDiskSpaceUsed.inc(newSize - oldSize); + // The new size will be added in Transactional.commit() as an updated SSTable, more details: CASSANDRA-13738 + StorageMetrics.load.dec(oldSize); + parent.metric.liveDiskSpaceUsed.dec(oldSize); + parent.metric.totalDiskSpaceUsed.dec(oldSize); return cloneAndReplace(first, OpenReason.METADATA_CHANGE, newSummary); } - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12793) invalid jvm type and architecture [cassandra-env.sh]
[ https://issues.apache.org/jira/browse/CASSANDRA-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153259#comment-16153259 ] Ali Ebrahiminejad commented on CASSANDRA-12793: --- {code:java} jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' | awk -F'"' 'NR==1 {print $2}' | cut -d\- -f1` JVM_VERSION=${jvmver%_*} JVM_PATCH_VERSION=${jvmver#*_} if [ "$JVM_VERSION" \< "1.8" ] ; then echo "Cassandra 3.0 and later require Java 8u40 or later." exit 1; fi if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" -lt 40 ] ; then echo "Cassandra 3.0 and later require Java 8u40 or later." exit 1; fi jvm=`echo "$java_ver_output" | grep -A 1 'java version' | awk 'NR==2 {print $1}'` {code} "jvmver" is just fine. The problem is with "jvm" which is a few lines below. > invalid jvm type and architecture [cassandra-env.sh] > > > Key: CASSANDRA-12793 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12793 > Project: Cassandra > Issue Type: Bug > Components: Configuration > Environment: ubuntu 16.04, openjdk 1.8.0_91 >Reporter: Ali Ebrahiminejad >Priority: Minor > Fix For: 3.7 > > > In cassandra-env.sh the part that determines the type of JVM we'll be running > on doesn't provide the right answer for openjdk 1.8.0_91. > value of java_ver_output is "openjdk version "1.8.0_91" OpenJDK Runtime > Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14) OpenJDK 64-Bit > Server VM (build 25.91-b14, mixed mode)", yet the command looks for "java > version" (jvm=`echo "$java_ver_output" | grep -A 1 'java version' ...) which > does not exist. > I guess it should be replaced with jvm=`echo "$java_ver_output" | grep -A 1 > '[openjdk|java] version' | awk 'NR==2 {print $1}'` -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-13657) Materialized Views: Index MV on TTL'ed column produces orphanized view entry if another column keeps entry live
[ https://issues.apache.org/jira/browse/CASSANDRA-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta resolved CASSANDRA-13657. - Resolution: Duplicate Fixed on CASSANDRA-11500, closing as duplicate. > Materialized Views: Index MV on TTL'ed column produces orphanized view entry > if another column keeps entry live > --- > > Key: CASSANDRA-13657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13657 > Project: Cassandra > Issue Type: Bug > Components: Materialized Views >Reporter: Fridtjof Sander >Assignee: Krishna Dattu Koneru > Labels: materializedviews, ttl > > {noformat} > CREATE TABLE t (k int, a int, b int, PRIMARY KEY (k)); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (a, k); > INSERT INTO t (k) VALUES (1); > UPDATE t USING TTL 5 SET a = 10 WHERE k = 1; > UPDATE t SET b = 100 WHERE k = 1; > SELECT * from t; SELECT * from mv; > k | a | b > ---++- > 1 | 10 | 100 > (1 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > -- 5 seconds later > SELECT * from t; SELECT * from mv; > k | a| b > ---+--+- > 1 | null | 100 > (1 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > -- that view entry's liveness-info is (probably) dead, but the entry is kept > alive by b=100 > DELETE b FROM t WHERE k=1; > SELECT * from t; SELECT * from mv; > k | a| b > ---+--+-- > 1 | null | null > (1 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > DELETE FROM t WHERE k=1; > cqlsh:test> SELECT * from t; SELECT * from mv; > k | a | b > ---+---+--- > (0 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > -- deleting the base-entry doesn't help, because the view-key can not be > constructed anymore (a=10 already expired) > {noformat} > The problem here is that although the view-entry's liveness-info (probably) > expired correctly a regular column (`b`) keeps the view-entry live. It should > have disappeared since it's indexed column (`a`) expired in the corresponding > base-row. This is pretty severe, since that view-entry is now orphanized. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13657) Materialized Views: Index MV on TTL'ed column produces orphanized view entry if another column keeps entry live
[ https://issues.apache.org/jira/browse/CASSANDRA-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-13657: Status: Open (was: Patch Available) > Materialized Views: Index MV on TTL'ed column produces orphanized view entry > if another column keeps entry live > --- > > Key: CASSANDRA-13657 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13657 > Project: Cassandra > Issue Type: Bug > Components: Materialized Views >Reporter: Fridtjof Sander >Assignee: Krishna Dattu Koneru > Labels: materializedviews, ttl > > {noformat} > CREATE TABLE t (k int, a int, b int, PRIMARY KEY (k)); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (a, k); > INSERT INTO t (k) VALUES (1); > UPDATE t USING TTL 5 SET a = 10 WHERE k = 1; > UPDATE t SET b = 100 WHERE k = 1; > SELECT * from t; SELECT * from mv; > k | a | b > ---++- > 1 | 10 | 100 > (1 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > -- 5 seconds later > SELECT * from t; SELECT * from mv; > k | a| b > ---+--+- > 1 | null | 100 > (1 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > -- that view entry's liveness-info is (probably) dead, but the entry is kept > alive by b=100 > DELETE b FROM t WHERE k=1; > SELECT * from t; SELECT * from mv; > k | a| b > ---+--+-- > 1 | null | null > (1 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > DELETE FROM t WHERE k=1; > cqlsh:test> SELECT * from t; SELECT * from mv; > k | a | b > ---+---+--- > (0 rows) > a | k | b > +---+- > 10 | 1 | 100 > (1 rows) > -- deleting the base-entry doesn't help, because the view-key can not be > constructed anymore (a=10 already expired) > {noformat} > The problem here is that although the view-entry's liveness-info (probably) > expired correctly a regular column (`b`) keeps the view-entry live. It should > have disappeared since it's indexed column (`a`) expired in the corresponding > base-row. This is pretty severe, since that view-entry is now orphanized. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[ https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153239#comment-16153239 ] Paulo Motta commented on CASSANDRA-11500: - bq. As we discussed offline, we need to make sure the raw data including tombstone, expired liveness are shipped to the coordinator side. Enforcing strict liveness in ReadCommand.executeLocally() would remove the row before digest or data response. Instead, we add enforceStrictLiveness to Row.purge to get the same result but less interfaces changes for Row. Excellent catch, my bad for overlooking this, but gladly you added a new dtest for it: [test_base_column_in_view_pk_complex_timestamp_with_flush|https://github.com/apache/cassandra-dtest/commit/6d77ace5361f020ba182072ade9f4ab98025c213#diff-62ba429edee6a4681782f078246c9893R993]. Enforcing strict liveness during reconciliation on the coordinator is the correct approach to differentiate between non-existing and removed rows via strict liveness. Ideally strict liveness would be a row property but as we saw in the previous version, this requires changes a bunch of other classes and interfaces, so let's keep it as a flag to Row.purge for now. bq. Shadowable tombstone will be deprecated and use expired livenessInfo if the deletion time is greater than merged-row deletion to avoid uncessary expired livenessInfo. LGTM, this will prevent using expired liveness info in most cases. We should probably remove this as part of CASSANDRA-13826 since we're overlading ExpiringLivenessInfo for a different purpose but it should be fine for now. Final patch and CI results look good and I'm confident we've exercised all edge cases properly with unit and dtests. Committed dtests to master as {{6d77ace5361f020ba182072ade9f4ab98025c213}}. Commited to cassandra-3.0 branch as {{1b36740ebe66b8ed4c3d6cb64eb2419a9279dfbf}} and merged up to cassandra-3.11 and trunk. Great job Zhao! *Summary of improvements and fixed issues* This patch address most outstanding timestamp problems with MVs on the 3.x series without changing the binary or storage protocol. The remaining issues which requires storage engine and binary protocol changes will be addressed on CASSANDRA-13826 on trunk. Below is a summary of the main changes and fixes this patch introduces. {noformat} * View same PK components as base * DELETE of selected column should not affect out of order updates (testPartialDeleteSelectedColumn) * DELETE of unselected column/collection should not affect ordered updates (testUpdateColumnNotInView, testPartialUpdateWithUnselectedCollections - CASSANDRA-13127) * Unselected columns should keep view row alive when other columns expire (testUnselectedColumnsTTL - CASSANDRA-13127) * Extra column on view PK * View row should expire when view PK column expires in base (testUpdateColumnInViewPKWithTTL - CASSANDRA-13657) * Commutative row deletion (testCommutativeRowDeletion, testCellTombstoneAndShadowableTombstones - CASSANDRA-13409) * Out of order updates to extra column on view PK (testUpdateWithColumnTimestampBiggerThanPk, testUpdateWithColumnTimestampSmallerThanPk, CASSANDRA-11500) * Unsupported scenarios * DELETE of unselected column should not affect out of order updates (testPartialDeleteUnselectedColumn - CASSANDRA-13127 - Added ) * Filtering by non-PK base column (Only on C* 3.11+, disallowed on CASSANDRA-13798) * Additional MV tests * testNonBaseColumnInViewPk * testRangeDeletion * testStrictLivenessTombstone * testFrozenCollectionsWithComplexInnerType * testMVWithDifferentColumns * testBaseTTLWithSameTimestampTest {noformat} *Summary of code changes* {noformat} 1. Using expired livenessInfo if computed deletion time is greater than merged row deletion. There are only 2 cases: a. non-pk base column used in view pk is removed by partial update or partial delete b. unselected base column is removed by partial update or partial delete Current shadowable tombstone is not used to avoid the issue of resurrecting deleted cells (CASSANDRA-13049). We will expired-livenessInfo and merged base row deletion instead. 2. It's strict-liveness iff there is non-key base column in view-pk. The existence of view row is solely base on this non-key base column. 3. If there is no non-base PK column in view-pk, the view's liveness/deletion is using max of base livenessIno + unselected column. unselected column's ttl is used only when it affects view row liveness. Selected columns won't contribute to livenessInfo or row deletion. * this wouldn't support complex cases as explained above. eg. c/d unselected, update c@10, delete c@11, update d@5. view row should be alive but dead - We added a note to NEWS.txt as well as documentation about this unsupported case. * since we change the way the deletion timestamp is computed in this case, an older deletion to the non-base
[jira] [Updated] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted
[ https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-11500: Resolution: Fixed Fix Version/s: (was: 3.11.x) (was: 4.x) (was: 3.0.x) 4.0 3.11.1 3.0.15 Status: Resolved (was: Patch Available) > Obsolete MV entry may not be properly deleted > - > > Key: CASSANDRA-11500 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11500 > Project: Cassandra > Issue Type: Bug > Components: Materialized Views >Reporter: Sylvain Lebresne >Assignee: ZhaoYang > Fix For: 3.0.15, 3.11.1, 4.0 > > > When a Materialized View uses a non-PK base table column in its PK, if an > update changes that column value, we add the new view entry and remove the > old one. When doing that removal, the current code uses the same timestamp > than for the liveness info of the new entry, which is the max timestamp for > any columns participating to the view PK. This is not correct for the > deletion as the old view entry could have other columns with higher timestamp > which won't be deleted as can easily shown by the failing of the following > test: > {noformat} > CREATE TABLE t (k int PRIMARY KEY, a int, b int); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (k, a); > INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0; > UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1; > UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; > SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old > (invalid) and the new one > {noformat} > So the correct timestamp to use for the deletion is the biggest timestamp in > the old view entry (which we know since we read the pre-existing base row), > and that is what CASSANDRA-11475 does (the test above thus doesn't fail on > that branch). > Unfortunately, even then we can still have problems if further updates > requires us to overide the old entry. Consider the following case: > {noformat} > CREATE TABLE t (k int PRIMARY KEY, a int, b int); > CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS > NOT NULL PRIMARY KEY (k, a); > INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0; > UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1; > UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the > entry for a=1 with timestamp 10 > UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert > an entry for a=1 but shouldn't be deleted by the prior deletion > UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this > game more than once > UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1; > ... > {noformat} > In a way, this is saying that the "shadowable" deletion mechanism is not > general enough: we need to be able to re-insert an entry when a prior one had > been deleted before, but we can't rely on timestamps being strictly bigger on > the re-insert. In that sense, this can be though as a similar problem than > CASSANDRA-10965, though the solution there of a single flag is not enough > since we can have to replace more than once. > I think the proper solution would be to ship enough information to always be > able to decide when a view deletion is shadowed. Which means that both > liveness info (for updates) and shadowable deletion would need to ship the > timestamp of any base table column that is part the view PK (so {{a}} in the > example below). It's doable (and not that hard really), but it does require > a change to the sstable and intra-node protocol, which makes this a bit > painful right now. > But I'll also note that as CASSANDRA-1096 shows, the timestamp is not even > enough since on equal timestamp the value can be the deciding factor. So in > theory we'd have to ship the value of those columns (in the case of a > deletion at least since we have it in the view PK for updates). That said, on > that last problem, my preference would be that we start prioritizing > CASSANDRA-6123 seriously so we don't have to care about conflicting timestamp > anymore, which would make this problem go away. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org