[jira] [Created] (CASSANDRA-13050) ReadCommand.CheckForAbort not monitoring CQL rows for range queries
Stefania created CASSANDRA-13050: Summary: ReadCommand.CheckForAbort not monitoring CQL rows for range queries Key: CASSANDRA-13050 URL: https://issues.apache.org/jira/browse/CASSANDRA-13050 Project: Cassandra Issue Type: Bug Components: Local Write-Read Paths Reporter: Stefania Assignee: Stefania Fix For: 3.x If I understood the iterator transformations introduced by CASSANDRA-9975 correctly, [ReadCommand.CheckForAbort|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/ReadCommand.java#L541] should apply itself before returning a partition. At the moment it is applied to row iterators for single command partitions, but for range queries I think it only monitors the query progress when a new partition is iterated, not when a new row is iterated. So for large partitions, we may fail to log a query as slow, or fail to abort it if the RPC timeout is exceeded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-13037) DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753581#comment-15753581 ] Stefania edited comment on CASSANDRA-13037 at 12/16/16 6:22 AM: Thanks for checking. I agree that the threads are not deadlocked, my language was imprecise. My best guess is that the non-periodic-tasks thread has not been signaled, or it has failed to get the correct segment offset before entering awaitUninterruptibly(). I'm multiplexing again [here|http://cassci.datastax.com/job/stef1927-testall-multiplex/59/], this time with a longer timeout (9.5 minutes rather than 5) and with some error log messages that are printed out if it takes longer than 1 minute to receive a signal in awaitDiskSync(), code is [here|https://github.com/apache/cassandra/compare/trunk...stef1927:13037-2.1#diff-7720d4b5123a354876e0b3139222f34eR643]. Normally the entire test succeedes in just over 2 minutes. was (Author: stefania): Thanks for checking. I agree that the threads are not deadlocked, my language was imprecise. My best guess is that the non-periodic-tasks thread has not been signaled, or it has failed to get the correct segment offset before entering awaitUninterruptibly(). I'm multiplexing again [here|http://cassci.datastax.com/job/stef1927-testall-multiplex/58/], this time with a longer timeout (9.5 minutes rather than 5) and with some error log messages that are printed out if it takes longer than 1 minute to receive a signal in awaitDiskSync(), code is [here|https://github.com/apache/cassandra/compare/trunk...stef1927:13037-2.1#diff-7720d4b5123a354876e0b3139222f34eR643]. Normally the entire test succeedes in just over 2 minutes. > DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2 > - > > Key: CASSANDRA-13037 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13037 > Project: Cassandra > Issue Type: Bug > Components: Testing >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x, 2.2.x > > > DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2: > http://cassci.datastax.com/job/cassandra-2.2_testall/589/testReport/junit/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/ > http://cassci.datastax.com/job/cassandra-2.1_testall/399/testReport/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/ > {code} > Error Message > Timeout occurred. Please note the time in the report does not reflect the > time until the timeout. > Stacktrace > junit.framework.AssertionFailedError: Timeout occurred. Please note the time > in the report does not reflect the time until the timeout. > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13037) DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753581#comment-15753581 ] Stefania commented on CASSANDRA-13037: -- Thanks for checking. I agree that the threads are not deadlocked, my language was imprecise. My best guess is that the non-periodic-tasks thread has not been signaled, or it has failed to get the correct segment offset before entering awaitUninterruptibly(). I'm multiplexing again [here|http://cassci.datastax.com/job/stef1927-testall-multiplex/58/], this time with a longer timeout (9.5 minutes rather than 5) and with some error log messages that are printed out if it takes longer than 1 minute to receive a signal in awaitDiskSync(), code is [here|https://github.com/apache/cassandra/compare/trunk...stef1927:13037-2.1#diff-7720d4b5123a354876e0b3139222f34eR643]. Normally the entire test succeedes in just over 2 minutes. > DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2 > - > > Key: CASSANDRA-13037 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13037 > Project: Cassandra > Issue Type: Bug > Components: Testing >Reporter: Stefania >Assignee: Stefania > Fix For: 2.1.x, 2.2.x > > > DropKeyspaceCommitLogRecycleTest.testRecycle times out in 2.1 and 2.2: > http://cassci.datastax.com/job/cassandra-2.2_testall/589/testReport/junit/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/ > http://cassci.datastax.com/job/cassandra-2.1_testall/399/testReport/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/ > {code} > Error Message > Timeout occurred. Please note the time in the report does not reflect the > time until the timeout. > Stacktrace > junit.framework.AssertionFailedError: Timeout occurred. Please note the time > in the report does not reflect the time until the timeout. > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-13049) Too many open files during bootstrapping
Simon Zhou created CASSANDRA-13049: -- Summary: Too many open files during bootstrapping Key: CASSANDRA-13049 URL: https://issues.apache.org/jira/browse/CASSANDRA-13049 Project: Cassandra Issue Type: Improvement Reporter: Simon Zhou Assignee: Simon Zhou We just upgraded from 2.2.5 to 3.0.10 and got issue during bootstrapping. So likely this is something made worse along with improving IO performance in Cassandra 3. On our side, the issue is that we have lots of small sstables and thus when bootstrapping a new node, it receives lots of files during streaming and Cassandra keeps all of them open for an unpredictable amount of time. Eventually we hit "Too many open files" error and around that time, I can see ~1M open files through lsof and almost all of them are *-Data.db and *-Index.db. Definitely we should use a better compaction strategy to reduce the number of sstables but I see a few possible improvements in Cassandra: 1. We use memory map when reading data from sstables. Every time we create a new memory map, there is one more file descriptor open. Memory map improves IO performance when dealing with large files, do we want to set a file size threshold when doing this? 2. Whenever we finished receiving a file from peer, we create a SSTableReader/BigTableReader, which includes opening the data file and index file, and keep them open until some time later (unpredictable). See StreamReceiveTask#L110, BigTableWriter#openFinal and SSTableReader#InstanceTidier. Is it better to lazily open the data/index files or close them more often to reclaim the file descriptors? I searched all known issue in JIRA and looks like this is a new issue in Cassandra 3. cc [~Stefania] for comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-13048) Support SASL mechanism negotiation in existing Authenticators
[ https://issues.apache.org/jira/browse/CASSANDRA-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Bromhead reassigned CASSANDRA-13048: Assignee: Ben Bromhead > Support SASL mechanism negotiation in existing Authenticators > - > > Key: CASSANDRA-13048 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13048 > Project: Cassandra > Issue Type: Sub-task >Reporter: Ben Bromhead >Assignee: Ben Bromhead >Priority: Minor > Labels: client-auth, client-impacting > > [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] adds > support for SASL mechanism negotitation to the native protocol. Existing > Authenticators should follow the SASL negotiation mechanism used in > [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] . > It may make sense to make the SASL negotiation mechanism extensible so other > Authenticators can use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-12704) snapshot build never be able to publish to mvn artifactory
[ https://issues.apache.org/jira/browse/CASSANDRA-12704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Zhuang reassigned CASSANDRA-12704: -- Assignee: Jay Zhuang > snapshot build never be able to publish to mvn artifactory > -- > > Key: CASSANDRA-12704 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12704 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.7 > > Attachments: 12704-trunk.txt > > > {code} > $ ant publish > {code} > works fine when property "release" is set, which publishes the binaries to > release Artifactory. > But for daily snapshot build, if "release" is set, it won't be snapshot build: > https://github.com/apache/cassandra/blob/cassandra-3.7/build.xml#L74 > if "release" is not set, it doesn't publish to snapshot Artifactory: > https://github.com/apache/cassandra/blob/cassandra-3.7/build.xml#L1888 > I would suggest just removing the "if check" for target "publish". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10145) Change protocol to allow sending key space independent of query string
[ https://issues.apache.org/jira/browse/CASSANDRA-10145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752652#comment-15752652 ] Sandeep Tamhankar commented on CASSANDRA-10145: --- Just confirmed that my branch (PYTHON-684) has been merged to master. We're ready to merge that down to cassandra-test and move things forward. I believe accepting this patch and merging the Python driver master to cassandra-test branch need to happen simultaneously. If the Python driver branch is merged down first, it will speak the "new" v5 against existing C* and dtests will fail. [~thobbs] I think you have the power in both repos, so I defer to you. > Change protocol to allow sending key space independent of query string > -- > > Key: CASSANDRA-10145 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10145 > Project: Cassandra > Issue Type: Improvement >Reporter: Vishy Kasar >Assignee: Sandeep Tamhankar > Fix For: 3.x > > Attachments: 10145-trunk.txt > > > Currently keyspace is either embedded in the query string or set through "use > keyspace" on a connection by client driver. > There are practical use cases where client user has query and keyspace > independently. In order for that scenario to work, they will have to create > one client session per keyspace or have to resort to some string replace > hackery. > It will be nice if protocol allowed sending keyspace separately from the > query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11596) Add native transport port to system.peers
[ https://issues.apache.org/jira/browse/CASSANDRA-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752495#comment-15752495 ] Ariel Weisberg commented on CASSANDRA-11596: I am doing this as part of CASSANDRA-7544 along with [~rkuris] > Add native transport port to system.peers > - > > Key: CASSANDRA-11596 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11596 > Project: Cassandra > Issue Type: Improvement > Components: Distributed Metadata >Reporter: Nico Haller >Assignee: Ariel Weisberg >Priority: Minor > Labels: lhf > > Is there any reason why the native transport port is not being stored in > system.peers along the rpc broadcast address and transmitted to the connected > drivers? > I would love to have that feature, that would allow me to "hide" my cluster > behind a reverse NAT or LB and only consume one external IP address and > forward packets based on the port the client is connecting to. > I guess it makes sense to provide the complete socket information instead of > just the address and using a default port setting on the client to complete > the connection information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-11596) Add native transport port to system.peers
[ https://issues.apache.org/jira/browse/CASSANDRA-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Weisberg reassigned CASSANDRA-11596: -- Assignee: Ariel Weisberg > Add native transport port to system.peers > - > > Key: CASSANDRA-11596 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11596 > Project: Cassandra > Issue Type: Improvement > Components: Distributed Metadata >Reporter: Nico Haller >Assignee: Ariel Weisberg >Priority: Minor > Labels: lhf > > Is there any reason why the native transport port is not being stored in > system.peers along the rpc broadcast address and transmitted to the connected > drivers? > I would love to have that feature, that would allow me to "hide" my cluster > behind a reverse NAT or LB and only consume one external IP address and > forward packets based on the port the client is connecting to. > I guess it makes sense to provide the complete socket information instead of > just the address and using a default port setting on the client to complete > the connection information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13048) Support SASL mechanism negotiation in existing Authenticators
[ https://issues.apache.org/jira/browse/CASSANDRA-13048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Bromhead updated CASSANDRA-13048: - Issue Type: Sub-task (was: Task) Parent: CASSANDRA-9362 > Support SASL mechanism negotiation in existing Authenticators > - > > Key: CASSANDRA-13048 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13048 > Project: Cassandra > Issue Type: Sub-task >Reporter: Ben Bromhead >Priority: Minor > Labels: client-auth, client-impacting > > [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] adds > support for SASL mechanism negotitation to the native protocol. Existing > Authenticators should follow the SASL negotiation mechanism used in > [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] . > It may make sense to make the SASL negotiation mechanism extensible so other > Authenticators can use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-13048) Support SASL mechanism negotiation in existing Authenticators
Ben Bromhead created CASSANDRA-13048: Summary: Support SASL mechanism negotiation in existing Authenticators Key: CASSANDRA-13048 URL: https://issues.apache.org/jira/browse/CASSANDRA-13048 Project: Cassandra Issue Type: Task Reporter: Ben Bromhead Priority: Minor [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] adds support for SASL mechanism negotitation to the native protocol. Existing Authenticators should follow the SASL negotiation mechanism used in [CASSANDRA-11471|https://issues.apache.org/jira/browse/CASSANDRA-11471] . It may make sense to make the SASL negotiation mechanism extensible so other Authenticators can use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12186) anticompaction log message doesn't include the parent repair session id
[ https://issues.apache.org/jira/browse/CASSANDRA-12186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-12186: Resolution: Fixed Fix Version/s: (was: 3.x) 3.10 3.0.11 Status: Resolved (was: Ready to Commit) > anticompaction log message doesn't include the parent repair session id > --- > > Key: CASSANDRA-12186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12186 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Wei Deng >Assignee: Tommy Stendahl >Priority: Minor > Labels: lhf > Fix For: 3.0.11, 3.10 > > Attachments: 12186.txt > > > It appears that even though incremental repair is now enabled by default post > C*-3.0 (which means at the end of each repair session, there is an > anti-compaction step that needs to be executed), we don't include the parent > repair session UUID in the log message of the anti-compaction log entries. > This makes observing all activities related to an incremental repair session > to be more difficult. See the following: > {noformat} > DEBUG [AntiEntropyStage:1] 2016-07-13 01:57:30,956 > RepairMessageVerbHandler.java:149 - Got anticompaction request > AnticompactionRequest{parentRepairSession=27103de0-489d-11e6-a6d6-cd06faa0aaa2} > org.apache.cassandra.repair.messages.AnticompactionRequest@34449ff4 > <...> > > <...> > INFO [CompactionExecutor:5] 2016-07-13 02:07:47,512 > CompactionManager.java:511 - Starting anticompaction for trivial_ks.weitest > on > 1/[BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')] > sstables > INFO [CompactionExecutor:5] 2016-07-13 02:07:47,513 > CompactionManager.java:540 - SSTable > BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db') > fully contained in range (-9223372036854775808,-9223372036854775808], > mutating repairedAt instead of anticompacting > INFO [CompactionExecutor:5] 2016-07-13 02:07:47,570 > CompactionManager.java:578 - Completed anticompaction successfully > {noformat} > The initial submission of the anti-compaction task to the CompactionManager > still has reference to the parent repair session UUID, but subsequent > anti-compaction log entries are missing this parent repair session UUID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12186) anticompaction log message doesn't include the parent repair session id
[ https://issues.apache.org/jira/browse/CASSANDRA-12186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752335#comment-15752335 ] Paulo Motta commented on CASSANDRA-12186: - Committed as 2256778726319fb76b6d85c4a47a957116c78147 on 3.0 and merged up. Thanks! > anticompaction log message doesn't include the parent repair session id > --- > > Key: CASSANDRA-12186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12186 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Wei Deng >Assignee: Tommy Stendahl >Priority: Minor > Labels: lhf > Fix For: 3.0.11, 3.10 > > Attachments: 12186.txt > > > It appears that even though incremental repair is now enabled by default post > C*-3.0 (which means at the end of each repair session, there is an > anti-compaction step that needs to be executed), we don't include the parent > repair session UUID in the log message of the anti-compaction log entries. > This makes observing all activities related to an incremental repair session > to be more difficult. See the following: > {noformat} > DEBUG [AntiEntropyStage:1] 2016-07-13 01:57:30,956 > RepairMessageVerbHandler.java:149 - Got anticompaction request > AnticompactionRequest{parentRepairSession=27103de0-489d-11e6-a6d6-cd06faa0aaa2} > org.apache.cassandra.repair.messages.AnticompactionRequest@34449ff4 > <...> > > <...> > INFO [CompactionExecutor:5] 2016-07-13 02:07:47,512 > CompactionManager.java:511 - Starting anticompaction for trivial_ks.weitest > on > 1/[BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')] > sstables > INFO [CompactionExecutor:5] 2016-07-13 02:07:47,513 > CompactionManager.java:540 - SSTable > BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db') > fully contained in range (-9223372036854775808,-9223372036854775808], > mutating repairedAt instead of anticompacting > INFO [CompactionExecutor:5] 2016-07-13 02:07:47,570 > CompactionManager.java:578 - Completed anticompaction successfully > {noformat} > The initial submission of the anti-compaction task to the CompactionManager > still has reference to the parent repair session UUID, but subsequent > anti-compaction log entries are missing this parent repair session UUID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[4/4] cassandra git commit: Merge branch 'cassandra-3.X' into trunk
Merge branch 'cassandra-3.X' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68b27b8c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68b27b8c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68b27b8c Branch: refs/heads/trunk Commit: 68b27b8c4090f6d04dd0b2f24e54f09e99168cf4 Parents: 7d266b9 fcc19a1 Author: Paulo MottaAuthored: Thu Dec 15 17:40:55 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 17:40:55 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/68b27b8c/CHANGES.txt -- diff --cc CHANGES.txt index fd888bc1,6cc177c..4506324 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -133,8 -123,9 +133,9 @@@ * Remove pre-startup check for open JMX port (CASSANDRA-12074) * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738) * Restore resumable hints delivery (CASSANDRA-11960) - * Properly report LWT contention (CASSANDRA-12626) + * Properly record CAS contention (CASSANDRA-12626) Merged from 3.0: + * Add parent repair session id to anticompaction log message (CASSANDRA-12186) * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) http://git-wip-us.apache.org/repos/asf/cassandra/blob/68b27b8c/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/68b27b8c/test/unit/org/apache/cassandra/db/compaction/AntiCompactionTest.java --
[2/4] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9924756d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9924756d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9924756d Branch: refs/heads/trunk Commit: 9924756da9c248c2365448998e63a36708188260 Parents: 489be96 2256778 Author: Paulo MottaAuthored: Thu Dec 15 17:40:12 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 17:40:12 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9924756d/CHANGES.txt -- diff --cc CHANGES.txt index fa0c94a,a40dabd..562b63d --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure strategies at the coordinator (CASSANDRA-9318) + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647) + * Fix cassandra-stress graphing (CASSANDRA-12237) + * Allow filtering on partition key columns for queries without secondary
[1/4] cassandra git commit: Add parent repair session id to anticompaction log message
Repository: cassandra Updated Branches: refs/heads/trunk 7d266b9e7 -> 68b27b8c4 Add parent repair session id to anticompaction log message Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787 Branch: refs/heads/trunk Commit: 2256778726319fb76b6d85c4a47a957116c78147 Parents: 48abc03 Author: tommy stendahlAuthored: Tue Nov 15 14:52:57 2016 +0100 Committer: Paulo Motta Committed: Thu Dec 15 17:38:20 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 63e095d..a40dabd 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Add parent repair session id to anticompaction log message (CASSANDRA-12186) * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index a77cefb..28140e0 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -456,7 +456,8 @@ public class CompactionManager implements CompactionManagerMBean public ListenableFuture submitAntiCompaction(final ColumnFamilyStore cfs, final Collection ranges, final Refs sstables, - final long repairedAt) + final long repairedAt, + final UUID parentRepairSession) { Runnable runnable = new WrappedRunnable() { @Override @@ -475,7 +476,7 @@ public class CompactionManager implements CompactionManagerMBean sstables.release(compactedSSTables); modifier = cfs.getTracker().tryModify(sstables, OperationType.ANTICOMPACTION); } -performAnticompaction(cfs, ranges, sstables, modifier, repairedAt); +performAnticompaction(cfs, ranges, sstables, modifier, repairedAt, parentRepairSession); } }; @@ -500,6 +501,7 @@ public class CompactionManager implements CompactionManagerMBean * @param cfs * @param ranges Ranges that the repair was carried out on * @param validatedForRepair SSTables containing the repaired ranges. Should be referenced before passing them. + * @param parentRepairSession parent repair session ID * @throws InterruptedException * @throws IOException */ @@ -507,10 +509,11 @@ public class CompactionManager implements CompactionManagerMBean Collection ranges, Refs validatedForRepair, LifecycleTransaction txn, - long repairedAt) throws InterruptedException, IOException + long repairedAt, + UUID parentRepairSession) throws InterruptedException, IOException { -logger.info("Starting anticompaction for {}.{} on {}/{} sstables", cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), cfs.getLiveSSTables()); -logger.trace("Starting anticompaction for ranges {}", ranges); +logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), validatedForRepair.size(), cfs.getLiveSSTables()); +logger.trace("[repair #{}] Starting anticompaction for ranges {}", parentRepairSession, ranges); Set sstables = new
[3/4] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X
Merge branch 'cassandra-3.11' into cassandra-3.X Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fcc19a1f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fcc19a1f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fcc19a1f Branch: refs/heads/trunk Commit: fcc19a1fa022dec02b8fe0c94e10973202c7bd50 Parents: b1e23ab 9924756 Author: Paulo MottaAuthored: Thu Dec 15 17:40:37 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 17:40:37 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/src/java/org/apache/cassandra/db/compaction/CompactionManager.java --
[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X
Merge branch 'cassandra-3.11' into cassandra-3.X Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fcc19a1f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fcc19a1f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fcc19a1f Branch: refs/heads/cassandra-3.X Commit: fcc19a1fa022dec02b8fe0c94e10973202c7bd50 Parents: b1e23ab 9924756 Author: Paulo MottaAuthored: Thu Dec 15 17:40:37 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 17:40:37 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/fcc19a1f/src/java/org/apache/cassandra/db/compaction/CompactionManager.java --
[1/3] cassandra git commit: Add parent repair session id to anticompaction log message
Repository: cassandra Updated Branches: refs/heads/cassandra-3.X b1e23ab54 -> fcc19a1fa Add parent repair session id to anticompaction log message Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787 Branch: refs/heads/cassandra-3.X Commit: 2256778726319fb76b6d85c4a47a957116c78147 Parents: 48abc03 Author: tommy stendahlAuthored: Tue Nov 15 14:52:57 2016 +0100 Committer: Paulo Motta Committed: Thu Dec 15 17:38:20 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 63e095d..a40dabd 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Add parent repair session id to anticompaction log message (CASSANDRA-12186) * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index a77cefb..28140e0 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -456,7 +456,8 @@ public class CompactionManager implements CompactionManagerMBean public ListenableFuture submitAntiCompaction(final ColumnFamilyStore cfs, final Collection ranges, final Refs sstables, - final long repairedAt) + final long repairedAt, + final UUID parentRepairSession) { Runnable runnable = new WrappedRunnable() { @Override @@ -475,7 +476,7 @@ public class CompactionManager implements CompactionManagerMBean sstables.release(compactedSSTables); modifier = cfs.getTracker().tryModify(sstables, OperationType.ANTICOMPACTION); } -performAnticompaction(cfs, ranges, sstables, modifier, repairedAt); +performAnticompaction(cfs, ranges, sstables, modifier, repairedAt, parentRepairSession); } }; @@ -500,6 +501,7 @@ public class CompactionManager implements CompactionManagerMBean * @param cfs * @param ranges Ranges that the repair was carried out on * @param validatedForRepair SSTables containing the repaired ranges. Should be referenced before passing them. + * @param parentRepairSession parent repair session ID * @throws InterruptedException * @throws IOException */ @@ -507,10 +509,11 @@ public class CompactionManager implements CompactionManagerMBean Collection ranges, Refs validatedForRepair, LifecycleTransaction txn, - long repairedAt) throws InterruptedException, IOException + long repairedAt, + UUID parentRepairSession) throws InterruptedException, IOException { -logger.info("Starting anticompaction for {}.{} on {}/{} sstables", cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), cfs.getLiveSSTables()); -logger.trace("Starting anticompaction for ranges {}", ranges); +logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), validatedForRepair.size(), cfs.getLiveSSTables()); +logger.trace("[repair #{}] Starting anticompaction for ranges {}", parentRepairSession, ranges);
[2/3] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9924756d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9924756d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9924756d Branch: refs/heads/cassandra-3.X Commit: 9924756da9c248c2365448998e63a36708188260 Parents: 489be96 2256778 Author: Paulo MottaAuthored: Thu Dec 15 17:40:12 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 17:40:12 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9924756d/CHANGES.txt -- diff --cc CHANGES.txt index fa0c94a,a40dabd..562b63d --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure strategies at the coordinator (CASSANDRA-9318) + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647) + * Fix cassandra-stress graphing (CASSANDRA-12237) + * Allow filtering on partition key columns for queries without
[1/2] cassandra git commit: Add parent repair session id to anticompaction log message
Repository: cassandra Updated Branches: refs/heads/cassandra-3.11 489be961c -> 9924756da Add parent repair session id to anticompaction log message Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787 Branch: refs/heads/cassandra-3.11 Commit: 2256778726319fb76b6d85c4a47a957116c78147 Parents: 48abc03 Author: tommy stendahlAuthored: Tue Nov 15 14:52:57 2016 +0100 Committer: Paulo Motta Committed: Thu Dec 15 17:38:20 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 63e095d..a40dabd 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Add parent repair session id to anticompaction log message (CASSANDRA-12186) * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index a77cefb..28140e0 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -456,7 +456,8 @@ public class CompactionManager implements CompactionManagerMBean public ListenableFuture submitAntiCompaction(final ColumnFamilyStore cfs, final Collection ranges, final Refs sstables, - final long repairedAt) + final long repairedAt, + final UUID parentRepairSession) { Runnable runnable = new WrappedRunnable() { @Override @@ -475,7 +476,7 @@ public class CompactionManager implements CompactionManagerMBean sstables.release(compactedSSTables); modifier = cfs.getTracker().tryModify(sstables, OperationType.ANTICOMPACTION); } -performAnticompaction(cfs, ranges, sstables, modifier, repairedAt); +performAnticompaction(cfs, ranges, sstables, modifier, repairedAt, parentRepairSession); } }; @@ -500,6 +501,7 @@ public class CompactionManager implements CompactionManagerMBean * @param cfs * @param ranges Ranges that the repair was carried out on * @param validatedForRepair SSTables containing the repaired ranges. Should be referenced before passing them. + * @param parentRepairSession parent repair session ID * @throws InterruptedException * @throws IOException */ @@ -507,10 +509,11 @@ public class CompactionManager implements CompactionManagerMBean Collection ranges, Refs validatedForRepair, LifecycleTransaction txn, - long repairedAt) throws InterruptedException, IOException + long repairedAt, + UUID parentRepairSession) throws InterruptedException, IOException { -logger.info("Starting anticompaction for {}.{} on {}/{} sstables", cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), cfs.getLiveSSTables()); -logger.trace("Starting anticompaction for ranges {}", ranges); +logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), validatedForRepair.size(), cfs.getLiveSSTables()); +logger.trace("[repair #{}] Starting anticompaction for ranges {}", parentRepairSession, ranges);
[2/2] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9924756d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9924756d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9924756d Branch: refs/heads/cassandra-3.11 Commit: 9924756da9c248c2365448998e63a36708188260 Parents: 489be96 2256778 Author: Paulo MottaAuthored: Thu Dec 15 17:40:12 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 17:40:12 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9924756d/CHANGES.txt -- diff --cc CHANGES.txt index fa0c94a,a40dabd..562b63d --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure strategies at the coordinator (CASSANDRA-9318) + * Make randompartitioner work with new vnode allocation (CASSANDRA-12647) + * Fix cassandra-stress graphing (CASSANDRA-12237) + * Allow filtering on partition key columns for queries without
cassandra git commit: Add parent repair session id to anticompaction log message
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 48abc0369 -> 225677872 Add parent repair session id to anticompaction log message Patch by Tommy Stendahl; Reviewed by Paulo Motta for CASSANDRA-12186 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22567787 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22567787 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22567787 Branch: refs/heads/cassandra-3.0 Commit: 2256778726319fb76b6d85c4a47a957116c78147 Parents: 48abc03 Author: tommy stendahlAuthored: Tue Nov 15 14:52:57 2016 +0100 Committer: Paulo Motta Committed: Thu Dec 15 17:38:20 2016 -0200 -- CHANGES.txt | 1 + .../db/compaction/CompactionManager.java| 21 +++- .../cassandra/service/ActiveRepairService.java | 4 ++-- .../db/compaction/AntiCompactionTest.java | 17 ++-- 4 files changed, 26 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 63e095d..a40dabd 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Add parent repair session id to anticompaction log message (CASSANDRA-12186) * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) http://git-wip-us.apache.org/repos/asf/cassandra/blob/22567787/src/java/org/apache/cassandra/db/compaction/CompactionManager.java -- diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java index a77cefb..28140e0 100644 --- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java +++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java @@ -456,7 +456,8 @@ public class CompactionManager implements CompactionManagerMBean public ListenableFuture submitAntiCompaction(final ColumnFamilyStore cfs, final Collection ranges, final Refs sstables, - final long repairedAt) + final long repairedAt, + final UUID parentRepairSession) { Runnable runnable = new WrappedRunnable() { @Override @@ -475,7 +476,7 @@ public class CompactionManager implements CompactionManagerMBean sstables.release(compactedSSTables); modifier = cfs.getTracker().tryModify(sstables, OperationType.ANTICOMPACTION); } -performAnticompaction(cfs, ranges, sstables, modifier, repairedAt); +performAnticompaction(cfs, ranges, sstables, modifier, repairedAt, parentRepairSession); } }; @@ -500,6 +501,7 @@ public class CompactionManager implements CompactionManagerMBean * @param cfs * @param ranges Ranges that the repair was carried out on * @param validatedForRepair SSTables containing the repaired ranges. Should be referenced before passing them. + * @param parentRepairSession parent repair session ID * @throws InterruptedException * @throws IOException */ @@ -507,10 +509,11 @@ public class CompactionManager implements CompactionManagerMBean Collection ranges, Refs validatedForRepair, LifecycleTransaction txn, - long repairedAt) throws InterruptedException, IOException + long repairedAt, + UUID parentRepairSession) throws InterruptedException, IOException { -logger.info("Starting anticompaction for {}.{} on {}/{} sstables", cfs.keyspace.getName(), cfs.getColumnFamilyName(), validatedForRepair.size(), cfs.getLiveSSTables()); -logger.trace("Starting anticompaction for ranges {}", ranges); +logger.info("[repair #{}] Starting anticompaction for {}.{} on {}/{} sstables", parentRepairSession, cfs.keyspace.getName(), cfs.getTableName(), validatedForRepair.size(), cfs.getLiveSSTables()); +logger.trace("[repair #{}] Starting anticompaction for ranges {}", parentRepairSession, ranges);
[jira] [Updated] (CASSANDRA-12905) Retry acquire MV lock on failure instead of throwing WTE on streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-12905: Resolution: Fixed Fix Version/s: 3.0.11 Status: Resolved (was: Patch Available) bq. If we have a hint file that is being processed and a WTE occurs in the middle, will the whole file be retransmitted or can it be resumed at the last successful position? It will resume at the latest successful page, not from the start of the hint file. bq. I guess this is not the case from my personal observations. I had situations with > 1GB hint queues per sender node which were not going to shrink due to WTEs. It seemed like the same hints have been retransmitted over and over again from scratch instead of trying to resume. Perhaps the hard-coded hint page size of 512KB is too coarse grained for cluster with MVs, what causes too much data to be re-transmitted on retry generating even more contention. We should perhaps make this configurable. bq. What helped in this situation was to pause hint delivery on all nodes but 1-2. I'm pretty sure the problem was WTEs due to lock contentions. To be honest, I did not try lowering hinted_handoff_throttle_in_kb or at least I don't remember. Yeah, lowering {{hinted_handoff_throttle_in_kb}} should help for clusters with MVs, and perhaps tunable hint page sizes too. You could maybe experiment with different values to see if you reach a more stable state. I committed the current work to 3.0 and up as 3faa0d925791be085b92949a0a0ec20f7e6ae368 and 48abc0369799acca0521150e2c88d4032e01c3b5 so we can unblock 3.10 release. Feel free to open follow-up tickets to further improve MV hints and streaming with the points we discussed here. Thanks! > Retry acquire MV lock on failure instead of throwing WTE on streaming > - > > Key: CASSANDRA-12905 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12905 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: centos 6.7 x86_64 >Reporter: Nir Zilka >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.0.11, 3.10 > > > Hello, > I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, > private VLAN), > first it was 2.2.5.1 and repair worked flawlessly, > second upgrade was to 3.0.9 (with upgradesstables) and also repair worked > well, > then i upgraded 2 weeks ago to 3.9 - and the repair problems started. > there are several errors types from the system.log (different nodes) : > - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx > - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation > timed out - received only 0 responses > - Remote peer xxx.xxx.xxx.xxx failed stream session > - Session completed with the following error > org.apache.cassandra.streaming.StreamException: Stream failed > > i use 3.9 default configuration with the cluster settings adjustments (3 > seeds, GossipingPropertyFileSnitch). > streaming_socket_timeout_in_ms is the default (8640). > i'm afraid from consistency problems while i'm not performing repair. > Any ideas? > Thanks, > Nir. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra-builds git commit: Fix triggerInterval variable scope
Repository: cassandra-builds Updated Branches: refs/heads/master dce81a6f3 -> 32939b643 Fix triggerInterval variable scope Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/32939b64 Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/32939b64 Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/32939b64 Branch: refs/heads/master Commit: 32939b643e22cc609fccae5d8d02f35cb754da02 Parents: dce81a6 Author: Michael ShulerAuthored: Thu Dec 15 13:19:44 2016 -0600 Committer: Michael Shuler Committed: Thu Dec 15 13:19:44 2016 -0600 -- jenkins-dsl/cassandra_job_dsl_seed.groovy | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/32939b64/jenkins-dsl/cassandra_job_dsl_seed.groovy -- diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy b/jenkins-dsl/cassandra_job_dsl_seed.groovy index be5bb0e..311f5e7 100644 --- a/jenkins-dsl/cassandra_job_dsl_seed.groovy +++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy @@ -226,9 +226,9 @@ cassandraBranches.each { // Run default dtest daily and variations weekly if (targetName == 'dtest') { -def triggerInterval = '@daily' +triggerInterval = '@daily' } else { -def triggerInterval = '@weekly' +triggerInterval = '@weekly' } // Skip dtest-offheap on cassandra-3.0 branch @@ -242,7 +242,7 @@ cassandraBranches.each { node / scm / branches / 'hudson.plugins.git.BranchSpec' / name(branchName) } triggers { -scm("${triggerInterval}") +scm(triggerInterval) } steps { shell("./cassandra-builds/build-scripts/cassandra-dtest.sh ${targetName}")
[4/5] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X
Merge branch 'cassandra-3.11' into cassandra-3.X Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b1e23ab5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b1e23ab5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b1e23ab5 Branch: refs/heads/trunk Commit: b1e23ab54f659755fdd7c1adf5c14b05088337f6 Parents: e5a77cf 489be96 Author: Paulo MottaAuthored: Thu Dec 15 16:50:30 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:50:30 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 89 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- src/java/org/apache/cassandra/hints/Hint.java | 40 ++--- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 6 +- 8 files changed, 90 insertions(+), 86 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b1e23ab5/CHANGES.txt --
[2/5] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock
Make hint delivery async so MV hint is deferred on failure to acquire lock Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036 Branch: refs/heads/trunk Commit: 48abc0369799acca0521150e2c88d4032e01c3b5 Parents: 3faa0d9 Author: Paulo MottaAuthored: Thu Dec 15 12:49:38 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:05 2016 -0200 -- src/java/org/apache/cassandra/hints/Hint.java | 40 ++-- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- 2 files changed, 30 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java -- diff --git a/src/java/org/apache/cassandra/hints/Hint.java b/src/java/org/apache/cassandra/hints/Hint.java index cbb5e74..17fbf5d 100644 --- a/src/java/org/apache/cassandra/hints/Hint.java +++ b/src/java/org/apache/cassandra/hints/Hint.java @@ -19,8 +19,12 @@ package org.apache.cassandra.hints; import java.io.IOException; import java.util.*; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; +import com.google.common.base.Throwables; + import org.apache.cassandra.db.*; import org.apache.cassandra.io.IVersionedSerializer; import org.apache.cassandra.io.util.DataInputPlus; @@ -68,7 +72,7 @@ public final class Hint return new Hint(mutation, creationTime, mutation.smallestGCGS()); } -/** +/* * @param mutation the hinted mutation * @param creationTime time of this hint's creation (in milliseconds since epoch) * @param gcgs the smallest gcgs of all tables involved at the time of hint creation (in seconds) @@ -81,19 +85,33 @@ public final class Hint /** * Applies the contained mutation unless it's expired, filtering out any updates for truncated tables */ -void apply() +CompletableFuture applyFuture() { -if (!isLive()) -return; +if (isLive()) +{ +// filter out partition update for table that have been truncated since hint's creation +Mutation filtered = mutation; +for (UUID id : mutation.getColumnFamilyIds()) +if (creationTime <= SystemKeyspace.getTruncatedAt(id)) +filtered = filtered.without(id); -// filter out partition update for table that have been truncated since hint's creation -Mutation filtered = mutation; -for (UUID id : mutation.getColumnFamilyIds()) -if (creationTime <= SystemKeyspace.getTruncatedAt(id)) -filtered = filtered.without(id); +if (!filtered.isEmpty()) +return filtered.applyFuture(); +} + +return CompletableFuture.completedFuture(null); +} -if (!filtered.isEmpty()) -filtered.apply(); +void apply() +{ +try +{ +applyFuture().get(); +} +catch (Exception e) +{ +throw Throwables.propagate(e.getCause()); +} } /** http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java -- diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java b/src/java/org/apache/cassandra/hints/HintVerbHandler.java index d8838a9..abcd1f9 100644 --- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java +++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java @@ -88,10 +88,8 @@ public final class HintVerbHandler implements IVerbHandler else { // the common path - the node is both the destination and a valid replica for the hint. -hint.apply(); +hint.applyFuture().thenAccept(o -> reply(id, message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); return null;}); } - -reply(id, message.from); } private static void reply(int id, InetAddress to)
[1/5] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming
Repository: cassandra Updated Branches: refs/heads/trunk 0f67b540e -> 7d266b9e7 Retry acquire MV lock on failure instead of throwing WTE on streaming Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92 Branch: refs/heads/trunk Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368 Parents: 9fc1ffb Author: brstgtAuthored: Thu Dec 15 12:42:31 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:00 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 92 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 5 +- 6 files changed, 63 insertions(+), 71 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e69bf08..63e095d 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java -- diff --git a/src/java/org/apache/cassandra/db/Keyspace.java b/src/java/org/apache/cassandra/db/Keyspace.java index ec5102b..3715995 100644 --- a/src/java/org/apache/cassandra/db/Keyspace.java +++ b/src/java/org/apache/cassandra/db/Keyspace.java @@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock; import com.google.common.base.Function; import com.google.common.collect.Iterables; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.apache.cassandra.concurrent.Stage; import org.apache.cassandra.concurrent.StageManager; @@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil; import org.apache.cassandra.utils.FBUtilities; import org.apache.cassandra.utils.JVMStabilityInspector; import org.apache.cassandra.utils.concurrent.OpOrder; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; /** * It represents a Keyspace. @@ -62,7 +62,7 @@ public class Keyspace private static final String TEST_FAIL_WRITES_KS = System.getProperty("cassandra.test.fail_writes_ks", ""); private static final boolean TEST_FAIL_WRITES = !TEST_FAIL_WRITES_KS.isEmpty(); -private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", "0"), 0); +private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger("cassandra.test.fail_mv_locks_count", 0); public final KeyspaceMetrics metric; @@ -379,42 +379,40 @@ public class Keyspace } } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog) -{ -return apply(mutation, writeCommitLog, true, false, null); -} - -/** - * Should be used if caller is blocking and runs in mutation stage. - * Otherwise there is a race condition where ALL mutation workers are beeing blocked ending - * in a complete deadlock of the mutation stage. See CASSANDRA-12689. - * - * @param mutation - * @param writeCommitLog - * @return - */ -public CompletableFuture applyNotDeferrable(Mutation mutation, boolean writeCommitLog) +public CompletableFuture applyFuture(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, true, false, false, null); +return apply(mutation, writeCommitLog, updateIndexes, true, true, null); } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) +public void apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, updateIndexes, false, null); +apply(mutation, writeCommitLog, updateIndexes, true); } -public CompletableFuture applyFromCommitLog(Mutation mutation) +public void apply(final Mutation mutation, + final boolean writeCommitLog)
[5/5] cassandra git commit: Merge branch 'cassandra-3.X' into trunk
Merge branch 'cassandra-3.X' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7d266b9e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7d266b9e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7d266b9e Branch: refs/heads/trunk Commit: 7d266b9e79e0738df25e4f86b5bdccb030a02d88 Parents: 0f67b54 b1e23ab Author: Paulo MottaAuthored: Thu Dec 15 16:51:12 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:51:12 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 89 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- src/java/org/apache/cassandra/hints/Hint.java | 40 ++--- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 6 +- 8 files changed, 90 insertions(+), 86 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d266b9e/CHANGES.txt -- diff --cc CHANGES.txt index e0c5637,5f7cf93..fd888bc1 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -133,8 -123,9 +133,9 @@@ * Remove pre-startup check for open JMX port (CASSANDRA-12074) * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738) * Restore resumable hints delivery (CASSANDRA-11960) - * Properly report LWT contention (CASSANDRA-12626) + * Properly record CAS contention (CASSANDRA-12626) Merged from 3.0: + * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7d266b9e/src/java/org/apache/cassandra/db/Mutation.java --
[3/5] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/489be961 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/489be961 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/489be961 Branch: refs/heads/trunk Commit: 489be961c945e4330a9426d21b2bb903cc1d3a54 Parents: 73547a3 48abc03 Author: Paulo MottaAuthored: Thu Dec 15 16:46:32 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:49:10 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 89 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- src/java/org/apache/cassandra/hints/Hint.java | 40 ++--- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 6 +- 8 files changed, 90 insertions(+), 86 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/489be961/CHANGES.txt -- diff --cc CHANGES.txt index 3db0179,63e095d..fa0c94a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure strategies at the coordinator
[4/4] cassandra git commit: Merge branch 'cassandra-3.11' into cassandra-3.X
Merge branch 'cassandra-3.11' into cassandra-3.X Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b1e23ab5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b1e23ab5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b1e23ab5 Branch: refs/heads/cassandra-3.X Commit: b1e23ab54f659755fdd7c1adf5c14b05088337f6 Parents: e5a77cf 489be96 Author: Paulo MottaAuthored: Thu Dec 15 16:50:30 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:50:30 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 89 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- src/java/org/apache/cassandra/hints/Hint.java | 40 ++--- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 6 +- 8 files changed, 90 insertions(+), 86 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b1e23ab5/CHANGES.txt --
[1/4] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming
Repository: cassandra Updated Branches: refs/heads/cassandra-3.X e5a77cfea -> b1e23ab54 Retry acquire MV lock on failure instead of throwing WTE on streaming Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92 Branch: refs/heads/cassandra-3.X Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368 Parents: 9fc1ffb Author: brstgtAuthored: Thu Dec 15 12:42:31 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:00 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 92 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 5 +- 6 files changed, 63 insertions(+), 71 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e69bf08..63e095d 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java -- diff --git a/src/java/org/apache/cassandra/db/Keyspace.java b/src/java/org/apache/cassandra/db/Keyspace.java index ec5102b..3715995 100644 --- a/src/java/org/apache/cassandra/db/Keyspace.java +++ b/src/java/org/apache/cassandra/db/Keyspace.java @@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock; import com.google.common.base.Function; import com.google.common.collect.Iterables; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.apache.cassandra.concurrent.Stage; import org.apache.cassandra.concurrent.StageManager; @@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil; import org.apache.cassandra.utils.FBUtilities; import org.apache.cassandra.utils.JVMStabilityInspector; import org.apache.cassandra.utils.concurrent.OpOrder; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; /** * It represents a Keyspace. @@ -62,7 +62,7 @@ public class Keyspace private static final String TEST_FAIL_WRITES_KS = System.getProperty("cassandra.test.fail_writes_ks", ""); private static final boolean TEST_FAIL_WRITES = !TEST_FAIL_WRITES_KS.isEmpty(); -private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", "0"), 0); +private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger("cassandra.test.fail_mv_locks_count", 0); public final KeyspaceMetrics metric; @@ -379,42 +379,40 @@ public class Keyspace } } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog) -{ -return apply(mutation, writeCommitLog, true, false, null); -} - -/** - * Should be used if caller is blocking and runs in mutation stage. - * Otherwise there is a race condition where ALL mutation workers are beeing blocked ending - * in a complete deadlock of the mutation stage. See CASSANDRA-12689. - * - * @param mutation - * @param writeCommitLog - * @return - */ -public CompletableFuture applyNotDeferrable(Mutation mutation, boolean writeCommitLog) +public CompletableFuture applyFuture(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, true, false, false, null); +return apply(mutation, writeCommitLog, updateIndexes, true, true, null); } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) +public void apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, updateIndexes, false, null); +apply(mutation, writeCommitLog, updateIndexes, true); } -public CompletableFuture applyFromCommitLog(Mutation mutation) +public void apply(final Mutation mutation, + final boolean
[3/4] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/489be961 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/489be961 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/489be961 Branch: refs/heads/cassandra-3.X Commit: 489be961c945e4330a9426d21b2bb903cc1d3a54 Parents: 73547a3 48abc03 Author: Paulo MottaAuthored: Thu Dec 15 16:46:32 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:49:10 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 89 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- src/java/org/apache/cassandra/hints/Hint.java | 40 ++--- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 6 +- 8 files changed, 90 insertions(+), 86 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/489be961/CHANGES.txt -- diff --cc CHANGES.txt index 3db0179,63e095d..fa0c94a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure strategies at the
[2/4] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock
Make hint delivery async so MV hint is deferred on failure to acquire lock Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036 Branch: refs/heads/cassandra-3.X Commit: 48abc0369799acca0521150e2c88d4032e01c3b5 Parents: 3faa0d9 Author: Paulo MottaAuthored: Thu Dec 15 12:49:38 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:05 2016 -0200 -- src/java/org/apache/cassandra/hints/Hint.java | 40 ++-- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- 2 files changed, 30 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java -- diff --git a/src/java/org/apache/cassandra/hints/Hint.java b/src/java/org/apache/cassandra/hints/Hint.java index cbb5e74..17fbf5d 100644 --- a/src/java/org/apache/cassandra/hints/Hint.java +++ b/src/java/org/apache/cassandra/hints/Hint.java @@ -19,8 +19,12 @@ package org.apache.cassandra.hints; import java.io.IOException; import java.util.*; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; +import com.google.common.base.Throwables; + import org.apache.cassandra.db.*; import org.apache.cassandra.io.IVersionedSerializer; import org.apache.cassandra.io.util.DataInputPlus; @@ -68,7 +72,7 @@ public final class Hint return new Hint(mutation, creationTime, mutation.smallestGCGS()); } -/** +/* * @param mutation the hinted mutation * @param creationTime time of this hint's creation (in milliseconds since epoch) * @param gcgs the smallest gcgs of all tables involved at the time of hint creation (in seconds) @@ -81,19 +85,33 @@ public final class Hint /** * Applies the contained mutation unless it's expired, filtering out any updates for truncated tables */ -void apply() +CompletableFuture applyFuture() { -if (!isLive()) -return; +if (isLive()) +{ +// filter out partition update for table that have been truncated since hint's creation +Mutation filtered = mutation; +for (UUID id : mutation.getColumnFamilyIds()) +if (creationTime <= SystemKeyspace.getTruncatedAt(id)) +filtered = filtered.without(id); -// filter out partition update for table that have been truncated since hint's creation -Mutation filtered = mutation; -for (UUID id : mutation.getColumnFamilyIds()) -if (creationTime <= SystemKeyspace.getTruncatedAt(id)) -filtered = filtered.without(id); +if (!filtered.isEmpty()) +return filtered.applyFuture(); +} + +return CompletableFuture.completedFuture(null); +} -if (!filtered.isEmpty()) -filtered.apply(); +void apply() +{ +try +{ +applyFuture().get(); +} +catch (Exception e) +{ +throw Throwables.propagate(e.getCause()); +} } /** http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java -- diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java b/src/java/org/apache/cassandra/hints/HintVerbHandler.java index d8838a9..abcd1f9 100644 --- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java +++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java @@ -88,10 +88,8 @@ public final class HintVerbHandler implements IVerbHandler else { // the common path - the node is both the destination and a valid replica for the hint. -hint.apply(); +hint.applyFuture().thenAccept(o -> reply(id, message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); return null;}); } - -reply(id, message.from); } private static void reply(int id, InetAddress to)
[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/489be961 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/489be961 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/489be961 Branch: refs/heads/cassandra-3.11 Commit: 489be961c945e4330a9426d21b2bb903cc1d3a54 Parents: 73547a3 48abc03 Author: Paulo MottaAuthored: Thu Dec 15 16:46:32 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:49:10 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 89 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- src/java/org/apache/cassandra/hints/Hint.java | 40 ++--- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 6 +- 8 files changed, 90 insertions(+), 86 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/489be961/CHANGES.txt -- diff --cc CHANGES.txt index 3db0179,63e095d..fa0c94a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure strategies at the
[2/3] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock
Make hint delivery async so MV hint is deferred on failure to acquire lock Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036 Branch: refs/heads/cassandra-3.11 Commit: 48abc0369799acca0521150e2c88d4032e01c3b5 Parents: 3faa0d9 Author: Paulo MottaAuthored: Thu Dec 15 12:49:38 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:05 2016 -0200 -- src/java/org/apache/cassandra/hints/Hint.java | 40 ++-- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- 2 files changed, 30 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java -- diff --git a/src/java/org/apache/cassandra/hints/Hint.java b/src/java/org/apache/cassandra/hints/Hint.java index cbb5e74..17fbf5d 100644 --- a/src/java/org/apache/cassandra/hints/Hint.java +++ b/src/java/org/apache/cassandra/hints/Hint.java @@ -19,8 +19,12 @@ package org.apache.cassandra.hints; import java.io.IOException; import java.util.*; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; +import com.google.common.base.Throwables; + import org.apache.cassandra.db.*; import org.apache.cassandra.io.IVersionedSerializer; import org.apache.cassandra.io.util.DataInputPlus; @@ -68,7 +72,7 @@ public final class Hint return new Hint(mutation, creationTime, mutation.smallestGCGS()); } -/** +/* * @param mutation the hinted mutation * @param creationTime time of this hint's creation (in milliseconds since epoch) * @param gcgs the smallest gcgs of all tables involved at the time of hint creation (in seconds) @@ -81,19 +85,33 @@ public final class Hint /** * Applies the contained mutation unless it's expired, filtering out any updates for truncated tables */ -void apply() +CompletableFuture applyFuture() { -if (!isLive()) -return; +if (isLive()) +{ +// filter out partition update for table that have been truncated since hint's creation +Mutation filtered = mutation; +for (UUID id : mutation.getColumnFamilyIds()) +if (creationTime <= SystemKeyspace.getTruncatedAt(id)) +filtered = filtered.without(id); -// filter out partition update for table that have been truncated since hint's creation -Mutation filtered = mutation; -for (UUID id : mutation.getColumnFamilyIds()) -if (creationTime <= SystemKeyspace.getTruncatedAt(id)) -filtered = filtered.without(id); +if (!filtered.isEmpty()) +return filtered.applyFuture(); +} + +return CompletableFuture.completedFuture(null); +} -if (!filtered.isEmpty()) -filtered.apply(); +void apply() +{ +try +{ +applyFuture().get(); +} +catch (Exception e) +{ +throw Throwables.propagate(e.getCause()); +} } /** http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java -- diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java b/src/java/org/apache/cassandra/hints/HintVerbHandler.java index d8838a9..abcd1f9 100644 --- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java +++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java @@ -88,10 +88,8 @@ public final class HintVerbHandler implements IVerbHandler else { // the common path - the node is both the destination and a valid replica for the hint. -hint.apply(); +hint.applyFuture().thenAccept(o -> reply(id, message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); return null;}); } - -reply(id, message.from); } private static void reply(int id, InetAddress to)
[1/3] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming
Repository: cassandra Updated Branches: refs/heads/cassandra-3.11 73547a342 -> 489be961c Retry acquire MV lock on failure instead of throwing WTE on streaming Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92 Branch: refs/heads/cassandra-3.11 Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368 Parents: 9fc1ffb Author: brstgtAuthored: Thu Dec 15 12:42:31 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:00 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 92 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 5 +- 6 files changed, 63 insertions(+), 71 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e69bf08..63e095d 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java -- diff --git a/src/java/org/apache/cassandra/db/Keyspace.java b/src/java/org/apache/cassandra/db/Keyspace.java index ec5102b..3715995 100644 --- a/src/java/org/apache/cassandra/db/Keyspace.java +++ b/src/java/org/apache/cassandra/db/Keyspace.java @@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock; import com.google.common.base.Function; import com.google.common.collect.Iterables; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.apache.cassandra.concurrent.Stage; import org.apache.cassandra.concurrent.StageManager; @@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil; import org.apache.cassandra.utils.FBUtilities; import org.apache.cassandra.utils.JVMStabilityInspector; import org.apache.cassandra.utils.concurrent.OpOrder; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; /** * It represents a Keyspace. @@ -62,7 +62,7 @@ public class Keyspace private static final String TEST_FAIL_WRITES_KS = System.getProperty("cassandra.test.fail_writes_ks", ""); private static final boolean TEST_FAIL_WRITES = !TEST_FAIL_WRITES_KS.isEmpty(); -private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", "0"), 0); +private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger("cassandra.test.fail_mv_locks_count", 0); public final KeyspaceMetrics metric; @@ -379,42 +379,40 @@ public class Keyspace } } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog) -{ -return apply(mutation, writeCommitLog, true, false, null); -} - -/** - * Should be used if caller is blocking and runs in mutation stage. - * Otherwise there is a race condition where ALL mutation workers are beeing blocked ending - * in a complete deadlock of the mutation stage. See CASSANDRA-12689. - * - * @param mutation - * @param writeCommitLog - * @return - */ -public CompletableFuture applyNotDeferrable(Mutation mutation, boolean writeCommitLog) +public CompletableFuture applyFuture(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, true, false, false, null); +return apply(mutation, writeCommitLog, updateIndexes, true, true, null); } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) +public void apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, updateIndexes, false, null); +apply(mutation, writeCommitLog, updateIndexes, true); } -public CompletableFuture applyFromCommitLog(Mutation mutation) +public void apply(final Mutation mutation, + final boolean
[2/2] cassandra git commit: Make hint delivery async so MV hint is deferred on failure to acquire lock
Make hint delivery async so MV hint is deferred on failure to acquire lock Patch by Paulo Motta; Reviewed by Benjamin Roth for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/48abc036 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/48abc036 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/48abc036 Branch: refs/heads/cassandra-3.0 Commit: 48abc0369799acca0521150e2c88d4032e01c3b5 Parents: 3faa0d9 Author: Paulo MottaAuthored: Thu Dec 15 12:49:38 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:05 2016 -0200 -- src/java/org/apache/cassandra/hints/Hint.java | 40 ++-- .../apache/cassandra/hints/HintVerbHandler.java | 4 +- 2 files changed, 30 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/Hint.java -- diff --git a/src/java/org/apache/cassandra/hints/Hint.java b/src/java/org/apache/cassandra/hints/Hint.java index cbb5e74..17fbf5d 100644 --- a/src/java/org/apache/cassandra/hints/Hint.java +++ b/src/java/org/apache/cassandra/hints/Hint.java @@ -19,8 +19,12 @@ package org.apache.cassandra.hints; import java.io.IOException; import java.util.*; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; +import com.google.common.base.Throwables; + import org.apache.cassandra.db.*; import org.apache.cassandra.io.IVersionedSerializer; import org.apache.cassandra.io.util.DataInputPlus; @@ -68,7 +72,7 @@ public final class Hint return new Hint(mutation, creationTime, mutation.smallestGCGS()); } -/** +/* * @param mutation the hinted mutation * @param creationTime time of this hint's creation (in milliseconds since epoch) * @param gcgs the smallest gcgs of all tables involved at the time of hint creation (in seconds) @@ -81,19 +85,33 @@ public final class Hint /** * Applies the contained mutation unless it's expired, filtering out any updates for truncated tables */ -void apply() +CompletableFuture applyFuture() { -if (!isLive()) -return; +if (isLive()) +{ +// filter out partition update for table that have been truncated since hint's creation +Mutation filtered = mutation; +for (UUID id : mutation.getColumnFamilyIds()) +if (creationTime <= SystemKeyspace.getTruncatedAt(id)) +filtered = filtered.without(id); -// filter out partition update for table that have been truncated since hint's creation -Mutation filtered = mutation; -for (UUID id : mutation.getColumnFamilyIds()) -if (creationTime <= SystemKeyspace.getTruncatedAt(id)) -filtered = filtered.without(id); +if (!filtered.isEmpty()) +return filtered.applyFuture(); +} + +return CompletableFuture.completedFuture(null); +} -if (!filtered.isEmpty()) -filtered.apply(); +void apply() +{ +try +{ +applyFuture().get(); +} +catch (Exception e) +{ +throw Throwables.propagate(e.getCause()); +} } /** http://git-wip-us.apache.org/repos/asf/cassandra/blob/48abc036/src/java/org/apache/cassandra/hints/HintVerbHandler.java -- diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java b/src/java/org/apache/cassandra/hints/HintVerbHandler.java index d8838a9..abcd1f9 100644 --- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java +++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java @@ -88,10 +88,8 @@ public final class HintVerbHandler implements IVerbHandler else { // the common path - the node is both the destination and a valid replica for the hint. -hint.apply(); +hint.applyFuture().thenAccept(o -> reply(id, message.from)).exceptionally(e -> {logger.debug("Failed to apply hint", e); return null;}); } - -reply(id, message.from); } private static void reply(int id, InetAddress to)
[1/2] cassandra git commit: Retry acquire MV lock on failure instead of throwing WTE on streaming
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 9fc1ffb63 -> 48abc0369 Retry acquire MV lock on failure instead of throwing WTE on streaming Patch by Benjamin Roth; Reviewed by Paulo Motta for CASSANDRA-12905 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3faa0d92 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3faa0d92 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3faa0d92 Branch: refs/heads/cassandra-3.0 Commit: 3faa0d925791be085b92949a0a0ec20f7e6ae368 Parents: 9fc1ffb Author: brstgtAuthored: Thu Dec 15 12:42:31 2016 -0200 Committer: Paulo Motta Committed: Thu Dec 15 16:46:00 2016 -0200 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/db/Keyspace.java | 92 +++- src/java/org/apache/cassandra/db/Mutation.java | 17 ++-- .../db/commitlog/CommitLogReplayer.java | 10 +-- .../cassandra/service/paxos/PaxosState.java | 9 +- .../cassandra/streaming/StreamReceiveTask.java | 5 +- 6 files changed, 63 insertions(+), 71 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e69bf08..63e095d 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Improve contention handling on failure to acquire MV lock for streaming and hints (CASSANDRA-12905) * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3faa0d92/src/java/org/apache/cassandra/db/Keyspace.java -- diff --git a/src/java/org/apache/cassandra/db/Keyspace.java b/src/java/org/apache/cassandra/db/Keyspace.java index ec5102b..3715995 100644 --- a/src/java/org/apache/cassandra/db/Keyspace.java +++ b/src/java/org/apache/cassandra/db/Keyspace.java @@ -26,6 +26,8 @@ import java.util.concurrent.locks.Lock; import com.google.common.base.Function; import com.google.common.collect.Iterables; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.apache.cassandra.concurrent.Stage; import org.apache.cassandra.concurrent.StageManager; @@ -50,8 +52,6 @@ import org.apache.cassandra.utils.ByteBufferUtil; import org.apache.cassandra.utils.FBUtilities; import org.apache.cassandra.utils.JVMStabilityInspector; import org.apache.cassandra.utils.concurrent.OpOrder; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; /** * It represents a Keyspace. @@ -62,7 +62,7 @@ public class Keyspace private static final String TEST_FAIL_WRITES_KS = System.getProperty("cassandra.test.fail_writes_ks", ""); private static final boolean TEST_FAIL_WRITES = !TEST_FAIL_WRITES_KS.isEmpty(); -private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger(System.getProperty("cassandra.test.fail_mv_locks_count", "0"), 0); +private static int TEST_FAIL_MV_LOCKS_COUNT = Integer.getInteger("cassandra.test.fail_mv_locks_count", 0); public final KeyspaceMetrics metric; @@ -379,42 +379,40 @@ public class Keyspace } } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog) -{ -return apply(mutation, writeCommitLog, true, false, null); -} - -/** - * Should be used if caller is blocking and runs in mutation stage. - * Otherwise there is a race condition where ALL mutation workers are beeing blocked ending - * in a complete deadlock of the mutation stage. See CASSANDRA-12689. - * - * @param mutation - * @param writeCommitLog - * @return - */ -public CompletableFuture applyNotDeferrable(Mutation mutation, boolean writeCommitLog) +public CompletableFuture applyFuture(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, true, false, false, null); +return apply(mutation, writeCommitLog, updateIndexes, true, true, null); } -public CompletableFuture apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) +public void apply(Mutation mutation, boolean writeCommitLog, boolean updateIndexes) { -return apply(mutation, writeCommitLog, updateIndexes, false, null); +apply(mutation, writeCommitLog, updateIndexes, true); } -public CompletableFuture applyFromCommitLog(Mutation mutation) +public void apply(final Mutation mutation, + final boolean
[jira] [Updated] (CASSANDRA-12510) Disallow decommission when number of replicas will drop below configured RF
[ https://issues.apache.org/jira/browse/CASSANDRA-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves updated CASSANDRA-12510: - Attachment: 12510-3.x-v2.patch Attached an updated version of the patch. Switched it back to cloneAfterAllLeft as I think I understand this a bit better now and there is no reason for settled. Also switched the {{isMember}} check back to the current tokenMetadata rather than the clone, this was just my misunderstanding of how things work before. And made the changes regarding the LEAVING state. I ended up including the whole for loop in the check (pending range check included) as I'm pretty sure it makes sense that a node can't have pending ranges while it's in the leaving state. [~pauloricardomg] I re-ran most of the failing tests and they seem OK now, not sure if I potentially broke other things though if you want to have a look :) > Disallow decommission when number of replicas will drop below configured RF > --- > > Key: CASSANDRA-12510 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12510 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging > Environment: C* version 3.3 >Reporter: Atin Sood >Assignee: Kurt Greaves >Priority: Minor > Labels: lhf > Attachments: 12510-3.x-v2.patch, 12510-3.x.patch > > > Steps to replicate : > - Create a 3 node cluster in DC1 and create a keyspace test_keyspace with > table test_table with replication strategy NetworkTopologyStrategy , DC1=3 . > Populate some data into this table. > - Add 5 more nodes to this cluster, but in DC2. Also do not alter the > keyspace to add the new DC2 to replication (this is intentional and the > reason why the bug shows up). So the desc keyspace should still list > NetworkTopologyStrategy with DC1=3 as RF > - As expected, this will now be a 8 node cluster with 3 nodes in DC1 and 5 in > DC2 > - Now start decommissioning the nodes in DC1. Note that the decommission runs > fine on all the 3 nodes, but since the new nodes are in DC2 and the RF for > keyspace is restricted to DC1, the new 5 nodes won't get any data. > - You will now end with the 5 node cluster which has no data from the > decommissioned 3 nodes and hence ending up in data loss > I do understand that this problem could have been avoided if we perform an > alter stmt and add DC2 replication before adding the 5 nodes. But the fact > that decommission ran fine on the 3 nodes on DC1 without complaining that > there were no nodes to stream its data seems a little discomforting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra-builds git commit: Pass DSL a varable string for interval
Repository: cassandra-builds Updated Branches: refs/heads/master 2a86477d5 -> dce81a6f3 Pass DSL a varable string for interval ERROR: (cassandra_job_dsl_seed.groovy, line 245) No such property: triggerInterval for class: javaposse.jobdsl.dsl.helpers.triggers.TriggerContext Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/dce81a6f Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/dce81a6f Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/dce81a6f Branch: refs/heads/master Commit: dce81a6f380559b13f3df48f350db6c82b3cfa11 Parents: 2a86477 Author: Michael ShulerAuthored: Thu Dec 15 13:01:15 2016 -0600 Committer: Michael Shuler Committed: Thu Dec 15 13:01:15 2016 -0600 -- jenkins-dsl/cassandra_job_dsl_seed.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/dce81a6f/jenkins-dsl/cassandra_job_dsl_seed.groovy -- diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy b/jenkins-dsl/cassandra_job_dsl_seed.groovy index 7bb30a4..be5bb0e 100644 --- a/jenkins-dsl/cassandra_job_dsl_seed.groovy +++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy @@ -242,7 +242,7 @@ cassandraBranches.each { node / scm / branches / 'hudson.plugins.git.BranchSpec' / name(branchName) } triggers { -scm(triggerInterval) +scm("${triggerInterval}") } steps { shell("./cassandra-builds/build-scripts/cassandra-dtest.sh ${targetName}")
cassandra-builds git commit: Run default dtest daily and variations weekly
Repository: cassandra-builds Updated Branches: refs/heads/master bb0105561 -> 2a86477d5 Run default dtest daily and variations weekly Project: http://git-wip-us.apache.org/repos/asf/cassandra-builds/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-builds/commit/2a86477d Tree: http://git-wip-us.apache.org/repos/asf/cassandra-builds/tree/2a86477d Diff: http://git-wip-us.apache.org/repos/asf/cassandra-builds/diff/2a86477d Branch: refs/heads/master Commit: 2a86477d5f7f73e6e9cd17468ea10679f078d81a Parents: bb01055 Author: Michael ShulerAuthored: Thu Dec 15 12:46:33 2016 -0600 Committer: Michael Shuler Committed: Thu Dec 15 12:46:33 2016 -0600 -- jenkins-dsl/cassandra_job_dsl_seed.groovy | 13 ++--- 1 file changed, 10 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra-builds/blob/2a86477d/jenkins-dsl/cassandra_job_dsl_seed.groovy -- diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy b/jenkins-dsl/cassandra_job_dsl_seed.groovy index 02bf485..7bb30a4 100644 --- a/jenkins-dsl/cassandra_job_dsl_seed.groovy +++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy @@ -150,9 +150,6 @@ job('Cassandra-template-dtest') { } } } -triggers { -scm('@daily') -} steps { buildDescription('', buildDescStr) shell("git clean -xdff ; git clone ${buildsRepo} ; git clone ${dtestRepo}") @@ -227,6 +224,13 @@ cassandraBranches.each { dtestTargets.each { def targetName = it +// Run default dtest daily and variations weekly +if (targetName == 'dtest') { +def triggerInterval = '@daily' +} else { +def triggerInterval = '@weekly' +} + // Skip dtest-offheap on cassandra-3.0 branch if ((targetName == 'dtest-offheap') && (branchName == 'cassandra-3.0')) { println("Skipping ${targetName} on branch ${branchName}") @@ -237,6 +241,9 @@ cassandraBranches.each { configure { node -> node / scm / branches / 'hudson.plugins.git.BranchSpec' / name(branchName) } +triggers { +scm(triggerInterval) +} steps { shell("./cassandra-builds/build-scripts/cassandra-dtest.sh ${targetName}") }
[jira] [Commented] (CASSANDRA-12947) Repair not replicating data
[ https://issues.apache.org/jira/browse/CASSANDRA-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752033#comment-15752033 ] Jeremiah Jordan commented on CASSANDRA-12947: - If you had to run scrub to fix some tables, did you run a repair -full after doing that? Scrub is going to throw away data, that has possibly already been repaired, so you need to do a repair -full to get it back. > Repair not replicating data > --- > > Key: CASSANDRA-12947 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12947 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.7 > Single DC > 7 Nodes > RF 3 > NetworkTopologyStrategy > OS: Ubuntu >Reporter: Malte Pickhan > Fix For: 3.x > > > We experienced strange behaviour of our C* cluster last week. > In the logs we've seen multiple requests in the logs where we the values read > from Cassandra actually have been 'null'. > When running the query on a node, we sometimes received the result and > sometimes not. > When setting the CL to LOCAL_QUORUM everything was fine. So far nothing > unusual, probably the dataset wasn't replicated to one of the nodes. > When turning on the tracing and running the query there was following > intersting line: > {quote} > Initiating read-repair [SharedPool-Worker-2] | 2016-11-18 10:17:47.528000 | > $PUBLIC_IP |126 | 127.0.0.1 > Digest mismatch: org.apache.cassandra.service.DigestMismatchException: > Mismatch for key DecoratedKey(-5887526567589486157, > 3130333031303338383436303937) (db1e86d507513ff12ba95f0eff984b60 vs > d41d8cd98f00b204e9800998ecf8427e) [ReadRepairStage:1] > {quote} > This is probably related to CASSANDRA-12090? > The interesting part is, after that we've run a 'nodetool repair -pr', after > that the behaviour was still the same and the data randomly not available, > depending on which node was hit. > Only after running a 'nodetool repair -pr -full' the issue was gone. > Did we miss something here? The point that's bothering me is that the dataset > was not replicated. > Worth to note is probably that some weeks ago we've hit the bug > CASSANDRA-12694 and fixed it by scrubbing some tables. > Any hints/help are appreciated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12829) DELETE query with an empty IN clause can delete more than expected
[ https://issues.apache.org/jira/browse/CASSANDRA-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-12829: --- Resolution: Fixed Fix Version/s: 3.10 3.0.11 Status: Resolved (was: Patch Available) > DELETE query with an empty IN clause can delete more than expected > -- > > Key: CASSANDRA-12829 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12829 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Arch Linux x64, kernel 4.7.6, Cassandra 3.9 downloaded > from the website >Reporter: Jason T. Bradshaw >Assignee: Alex Petrov > Fix For: 3.0.11, 3.10 > > > When deleting from a table with a certain structure and using an *in* clause > with an empty list, the *in* clause with an empty list can be ignored, > resulting in deleting more than is expected. > *Setup:* > {code} > cqlsh> create table test (a text, b text, id uuid, primary key ((a, b), id)); > cqlsh> insert into test (a, b, id) values ('a', 'b', > ----); > cqlsh> insert into test (a, b, id) values ('b', 'c', > ----); > cqlsh> insert into test (a, b, id) values ('a', 'c', > ----); > cqlsh> select * from test; > a | b | id > ---+---+-- > a | c | ---- > b | c | ---- > a | b | ---- > (3 rows) > {code} > *Expected:* > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in (); > cqlsh> select * from test; > a | b | id > ---+---+-- > a | c | ---- > b | c | ---- > a | b | ---- > (3 rows) > {code} > *Actual:* > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in (); > cqlsh> select * from test; > a | b | id > ---+---+-- > b | c | ---- > (1 rows) > {code} > Instead of deleting nothing, as the final empty *in* clause would imply, it > instead deletes everything that matches the first two clauses, acting as if > the following query had been issued instead: > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c'); > {code} > This seems to be related to the presence of a tuple clustering key, as I > could not reproduce it without one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12829) DELETE query with an empty IN clause can delete more than expected
[ https://issues.apache.org/jira/browse/CASSANDRA-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751913#comment-15751913 ] Benjamin Lerer commented on CASSANDRA-12829: Thanks for the patch. Committed into 3.0 at 9fc1ffb631a5a44bc11b55061288758e39a9d20a and merged into, 3.11, 3.X and trunk > DELETE query with an empty IN clause can delete more than expected > -- > > Key: CASSANDRA-12829 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12829 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Arch Linux x64, kernel 4.7.6, Cassandra 3.9 downloaded > from the website >Reporter: Jason T. Bradshaw >Assignee: Alex Petrov > > When deleting from a table with a certain structure and using an *in* clause > with an empty list, the *in* clause with an empty list can be ignored, > resulting in deleting more than is expected. > *Setup:* > {code} > cqlsh> create table test (a text, b text, id uuid, primary key ((a, b), id)); > cqlsh> insert into test (a, b, id) values ('a', 'b', > ----); > cqlsh> insert into test (a, b, id) values ('b', 'c', > ----); > cqlsh> insert into test (a, b, id) values ('a', 'c', > ----); > cqlsh> select * from test; > a | b | id > ---+---+-- > a | c | ---- > b | c | ---- > a | b | ---- > (3 rows) > {code} > *Expected:* > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in (); > cqlsh> select * from test; > a | b | id > ---+---+-- > a | c | ---- > b | c | ---- > a | b | ---- > (3 rows) > {code} > *Actual:* > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in (); > cqlsh> select * from test; > a | b | id > ---+---+-- > b | c | ---- > (1 rows) > {code} > Instead of deleting nothing, as the final empty *in* clause would imply, it > instead deletes everything that matches the first two clauses, acting as if > the following query had been issued instead: > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c'); > {code} > This seems to be related to the presence of a tuple clustering key, as I > could not reproduce it without one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/4] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11
Merge branch cassandra-3.0 into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/73547a34 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/73547a34 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/73547a34 Branch: refs/heads/trunk Commit: 73547a3421cc92e07d0373846fc1bff796570275 Parents: 249e4b1 9fc1ffb Author: Benjamin LererAuthored: Thu Dec 15 18:04:34 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:07:59 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 +++ .../cql3/statements/BatchStatement.java | 11 ++- .../cql3/statements/ModificationStatement.java | 10 ++- .../cql3/validation/operations/BatchTest.java | 40 +++ .../cql3/validation/operations/DeleteTest.java | 57 .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 70 8 files changed, 194 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/73547a34/CHANGES.txt -- diff --cc CHANGES.txt index f95dd81,e69bf08..3db0179 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure strategies
[4/4] cassandra git commit: Merge branch cassandra-3.X into trunk
Merge branch cassandra-3.X into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0f67b540 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0f67b540 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0f67b540 Branch: refs/heads/trunk Commit: 0f67b540e8e2c022348e8c163879d433e8d18fc3 Parents: b88ddb8 e5a77cf Author: Benjamin LererAuthored: Thu Dec 15 18:14:44 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:14:56 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 +++ .../cql3/statements/BatchStatement.java | 11 ++- .../cql3/statements/ModificationStatement.java | 10 ++- .../cql3/validation/operations/BatchTest.java | 40 +++ .../cql3/validation/operations/DeleteTest.java | 57 .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 70 8 files changed, 194 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/CHANGES.txt -- diff --cc CHANGES.txt index 9e7dd80,d32d91c..e0c5637 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -133,8 -123,9 +133,9 @@@ * Remove pre-startup check for open JMX port (CASSANDRA-12074) * Remove compaction Severity from DynamicEndpointSnitch (CASSANDRA-11738) * Restore resumable hints delivery (CASSANDRA-11960) - * Properly report LWT contention (CASSANDRA-12626) + * Properly record CAS contention (CASSANDRA-12626) Merged from 3.0: + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021) http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f67b540/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java --
[1/4] cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions
Repository: cassandra Updated Branches: refs/heads/trunk b88ddb8ab -> 0f67b540e Fix DELETE and UPDATE queries with empty IN restrictions patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6 Branch: refs/heads/trunk Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a Parents: 0fe82be Author: Alex PetrovAuthored: Thu Dec 15 17:56:41 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:01:32 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 .../cql3/statements/BatchStatement.java | 11 ++-- .../cql3/statements/ModificationStatement.java | 15 +++-- .../org/apache/cassandra/cql3/CQLTester.java| 7 ++ .../cql3/validation/operations/BatchTest.java | 40 .../cql3/validation/operations/DeleteTest.java | 63 -- .../cql3/validation/operations/InsertTest.java | 6 -- .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 69 +++- 10 files changed, 202 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 145afb9..e69bf08 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java -- diff --git a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java index 647d22f..542dec9 100644 --- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java +++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java @@ -430,6 +430,17 @@ public final class StatementRestrictions } /** + * Checks if restrictions on the clustering key have IN restrictions. + * + * @return true if the restrictions on the clustering key have IN restrictions, + * false otherwise. + */ +public boolean clusteringKeyRestrictionsHasIN() +{ +return clusteringColumnsRestrictions.isIN(); +} + +/** * Processes the clustering column restrictions. * * @param hasQueriableIndex true if some of the queried data are indexed, false otherwise http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java index f0aa835..76a6460 100644 --- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java @@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement QueryOptions statementOptions = options.forStatement(i); long timestamp = attrs.getTimestamp(now, statementOptions); List pks = statement.buildPartitionKeyNames(statementOptions); -if (pks.size() > 1) +if (statement.getRestrictions().keyIsInRelation()) throw new IllegalArgumentException("Batch with conditions cannot span multiple partitions (you cannot use IN on the partition key)"); if (key == null) { @@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement throw new InvalidRequestException("Batch with conditions cannot span multiple partitions"); } -SortedSet clusterings = statement.createClustering(statementOptions); + checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(), + "IN on the clustering key columns is not supported with conditional %s", + statement.type.isUpdate()? "updates" : "deletions"); -
[3/4] cassandra git commit: Merge branch cassandra-3.11 into cassandra-3.X
Merge branch cassandra-3.11 into cassandra-3.X Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e5a77cfe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e5a77cfe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e5a77cfe Branch: refs/heads/trunk Commit: e5a77cfeaa2be7f6c68965603e7a87034ff7b576 Parents: de84f26 73547a3 Author: Benjamin LererAuthored: Thu Dec 15 18:09:52 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:10:01 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 +++ .../cql3/statements/BatchStatement.java | 11 ++- .../cql3/statements/ModificationStatement.java | 10 ++- .../cql3/validation/operations/BatchTest.java | 40 +++ .../cql3/validation/operations/DeleteTest.java | 57 .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 70 8 files changed, 194 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java --
[3/3] cassandra git commit: Merge branch cassandra-3.11 into cassandra-3.X
Merge branch cassandra-3.11 into cassandra-3.X Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e5a77cfe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e5a77cfe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e5a77cfe Branch: refs/heads/cassandra-3.X Commit: e5a77cfeaa2be7f6c68965603e7a87034ff7b576 Parents: de84f26 73547a3 Author: Benjamin LererAuthored: Thu Dec 15 18:09:52 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:10:01 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 +++ .../cql3/statements/BatchStatement.java | 11 ++- .../cql3/statements/ModificationStatement.java | 10 ++- .../cql3/validation/operations/BatchTest.java | 40 +++ .../cql3/validation/operations/DeleteTest.java | 57 .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 70 8 files changed, 194 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e5a77cfe/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java --
[1/3] cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions
Repository: cassandra Updated Branches: refs/heads/cassandra-3.X de84f264f -> e5a77cfea Fix DELETE and UPDATE queries with empty IN restrictions patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6 Branch: refs/heads/cassandra-3.X Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a Parents: 0fe82be Author: Alex PetrovAuthored: Thu Dec 15 17:56:41 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:01:32 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 .../cql3/statements/BatchStatement.java | 11 ++-- .../cql3/statements/ModificationStatement.java | 15 +++-- .../org/apache/cassandra/cql3/CQLTester.java| 7 ++ .../cql3/validation/operations/BatchTest.java | 40 .../cql3/validation/operations/DeleteTest.java | 63 -- .../cql3/validation/operations/InsertTest.java | 6 -- .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 69 +++- 10 files changed, 202 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 145afb9..e69bf08 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java -- diff --git a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java index 647d22f..542dec9 100644 --- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java +++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java @@ -430,6 +430,17 @@ public final class StatementRestrictions } /** + * Checks if restrictions on the clustering key have IN restrictions. + * + * @return true if the restrictions on the clustering key have IN restrictions, + * false otherwise. + */ +public boolean clusteringKeyRestrictionsHasIN() +{ +return clusteringColumnsRestrictions.isIN(); +} + +/** * Processes the clustering column restrictions. * * @param hasQueriableIndex true if some of the queried data are indexed, false otherwise http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java index f0aa835..76a6460 100644 --- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java @@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement QueryOptions statementOptions = options.forStatement(i); long timestamp = attrs.getTimestamp(now, statementOptions); List pks = statement.buildPartitionKeyNames(statementOptions); -if (pks.size() > 1) +if (statement.getRestrictions().keyIsInRelation()) throw new IllegalArgumentException("Batch with conditions cannot span multiple partitions (you cannot use IN on the partition key)"); if (key == null) { @@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement throw new InvalidRequestException("Batch with conditions cannot span multiple partitions"); } -SortedSet clusterings = statement.createClustering(statementOptions); + checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(), + "IN on the clustering key columns is not supported with conditional %s", + statement.type.isUpdate()? "updates" : "deletions"); -
[2/3] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11
Merge branch cassandra-3.0 into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/73547a34 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/73547a34 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/73547a34 Branch: refs/heads/cassandra-3.X Commit: 73547a3421cc92e07d0373846fc1bff796570275 Parents: 249e4b1 9fc1ffb Author: Benjamin LererAuthored: Thu Dec 15 18:04:34 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:07:59 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 +++ .../cql3/statements/BatchStatement.java | 11 ++- .../cql3/statements/ModificationStatement.java | 10 ++- .../cql3/validation/operations/BatchTest.java | 40 +++ .../cql3/validation/operations/DeleteTest.java | 57 .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 70 8 files changed, 194 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/73547a34/CHANGES.txt -- diff --cc CHANGES.txt index f95dd81,e69bf08..3db0179 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure
[1/2] cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions
Repository: cassandra Updated Branches: refs/heads/cassandra-3.11 249e4b195 -> 73547a342 Fix DELETE and UPDATE queries with empty IN restrictions patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6 Branch: refs/heads/cassandra-3.11 Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a Parents: 0fe82be Author: Alex PetrovAuthored: Thu Dec 15 17:56:41 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:01:32 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 .../cql3/statements/BatchStatement.java | 11 ++-- .../cql3/statements/ModificationStatement.java | 15 +++-- .../org/apache/cassandra/cql3/CQLTester.java| 7 ++ .../cql3/validation/operations/BatchTest.java | 40 .../cql3/validation/operations/DeleteTest.java | 63 -- .../cql3/validation/operations/InsertTest.java | 6 -- .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 69 +++- 10 files changed, 202 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 145afb9..e69bf08 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java -- diff --git a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java index 647d22f..542dec9 100644 --- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java +++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java @@ -430,6 +430,17 @@ public final class StatementRestrictions } /** + * Checks if restrictions on the clustering key have IN restrictions. + * + * @return true if the restrictions on the clustering key have IN restrictions, + * false otherwise. + */ +public boolean clusteringKeyRestrictionsHasIN() +{ +return clusteringColumnsRestrictions.isIN(); +} + +/** * Processes the clustering column restrictions. * * @param hasQueriableIndex true if some of the queried data are indexed, false otherwise http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java index f0aa835..76a6460 100644 --- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java @@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement QueryOptions statementOptions = options.forStatement(i); long timestamp = attrs.getTimestamp(now, statementOptions); List pks = statement.buildPartitionKeyNames(statementOptions); -if (pks.size() > 1) +if (statement.getRestrictions().keyIsInRelation()) throw new IllegalArgumentException("Batch with conditions cannot span multiple partitions (you cannot use IN on the partition key)"); if (key == null) { @@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement throw new InvalidRequestException("Batch with conditions cannot span multiple partitions"); } -SortedSet clusterings = statement.createClustering(statementOptions); + checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(), + "IN on the clustering key columns is not supported with conditional %s", + statement.type.isUpdate()? "updates" : "deletions"); -
[2/2] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11
Merge branch cassandra-3.0 into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/73547a34 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/73547a34 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/73547a34 Branch: refs/heads/cassandra-3.11 Commit: 73547a3421cc92e07d0373846fc1bff796570275 Parents: 249e4b1 9fc1ffb Author: Benjamin LererAuthored: Thu Dec 15 18:04:34 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:07:59 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 +++ .../cql3/statements/BatchStatement.java | 11 ++- .../cql3/statements/ModificationStatement.java | 10 ++- .../cql3/validation/operations/BatchTest.java | 40 +++ .../cql3/validation/operations/DeleteTest.java | 57 .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 70 8 files changed, 194 insertions(+), 12 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/73547a34/CHANGES.txt -- diff --cc CHANGES.txt index f95dd81,e69bf08..3db0179 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,114 -1,5 +1,115 @@@ -3.0.11 +3.10 + * Remove outboundBindAny configuration property (CASSANDRA-12673) + * Use correct bounds for all-data range when filtering (CASSANDRA-12666) + * Remove timing window in test case (CASSANDRA-12875) + * Resolve unit testing without JCE security libraries installed (CASSANDRA-12945) + * Fix inconsistencies in cassandra-stress load balancing policy (CASSANDRA-12919) + * Fix validation of non-frozen UDT cells (CASSANDRA-12916) + * Don't shut down socket input/output on StreamSession (CASSANDRA-12903) + * Fix Murmur3PartitionerTest (CASSANDRA-12858) + * Move cqlsh syntax rules into separate module and allow easier customization (CASSANDRA-12897) + * Fix CommitLogSegmentManagerTest (CASSANDRA-12283) + * Fix cassandra-stress truncate option (CASSANDRA-12695) + * Fix crossNode value when receiving messages (CASSANDRA-12791) + * Don't load MX4J beans twice (CASSANDRA-12869) + * Extend native protocol request flags, add versions to SUPPORTED, and introduce ProtocolVersion enum (CASSANDRA-12838) + * Set JOINING mode when running pre-join tasks (CASSANDRA-12836) + * remove net.mintern.primitive library due to license issue (CASSANDRA-12845) + * Properly format IPv6 addresses when logging JMX service URL (CASSANDRA-12454) + * Optimize the vnode allocation for single replica per DC (CASSANDRA-12777) + * Use non-token restrictions for bounds when token restrictions are overridden (CASSANDRA-12419) + * Fix CQLSH auto completion for PER PARTITION LIMIT (CASSANDRA-12803) + * Use different build directories for Eclipse and Ant (CASSANDRA-12466) + * Avoid potential AttributeError in cqlsh due to no table metadata (CASSANDRA-12815) + * Fix RandomReplicationAwareTokenAllocatorTest.testExistingCluster (CASSANDRA-12812) + * Upgrade commons-codec to 1.9 (CASSANDRA-12790) + * Make the fanout size for LeveledCompactionStrategy to be configurable (CASSANDRA-11550) + * Add duration data type (CASSANDRA-11873) + * Fix timeout in ReplicationAwareTokenAllocatorTest (CASSANDRA-12784) + * Improve sum aggregate functions (CASSANDRA-12417) + * Make cassandra.yaml docs for batch_size_*_threshold_in_kb reflect changes in CASSANDRA-10876 (CASSANDRA-12761) + * cqlsh fails to format collections when using aliases (CASSANDRA-11534) + * Check for hash conflicts in prepared statements (CASSANDRA-12733) + * Exit query parsing upon first error (CASSANDRA-12598) + * Fix cassandra-stress to use single seed in UUID generation (CASSANDRA-12729) + * CQLSSTableWriter does not allow Update statement (CASSANDRA-12450) + * Config class uses boxed types but DD exposes primitive types (CASSANDRA-12199) + * Add pre- and post-shutdown hooks to Storage Service (CASSANDRA-12461) + * Add hint delivery metrics (CASSANDRA-12693) + * Remove IndexInfo cache from FileIndexInfoRetriever (CASSANDRA-12731) + * ColumnIndex does not reuse buffer (CASSANDRA-12502) + * cdc column addition still breaks schema migration tasks (CASSANDRA-12697) + * Upgrade metrics-reporter dependencies (CASSANDRA-12089) + * Tune compaction thread count via nodetool (CASSANDRA-12248) + * Add +=/-= shortcut syntax for update queries (CASSANDRA-12232) + * Include repair session IDs in repair start message (CASSANDRA-12532) + * Add a blocking task to Index, run before joining the ring (CASSANDRA-12039) + * Fix NPE when using CQLSSTableWriter (CASSANDRA-12667) + * Support optional backpressure
cassandra git commit: Fix DELETE and UPDATE queries with empty IN restrictions
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 0fe82be83 -> 9fc1ffb63 Fix DELETE and UPDATE queries with empty IN restrictions patch by Alex Petrov; reviewed by Benjamin Lerer for CASSANDRA-12829 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9fc1ffb6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9fc1ffb6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9fc1ffb6 Branch: refs/heads/cassandra-3.0 Commit: 9fc1ffb631a5a44bc11b55061288758e39a9d20a Parents: 0fe82be Author: Alex PetrovAuthored: Thu Dec 15 17:56:41 2016 +0100 Committer: Benjamin Lerer Committed: Thu Dec 15 18:01:32 2016 +0100 -- CHANGES.txt | 1 + .../restrictions/StatementRestrictions.java | 11 .../cql3/statements/BatchStatement.java | 11 ++-- .../cql3/statements/ModificationStatement.java | 15 +++-- .../org/apache/cassandra/cql3/CQLTester.java| 7 ++ .../cql3/validation/operations/BatchTest.java | 40 .../cql3/validation/operations/DeleteTest.java | 63 -- .../cql3/validation/operations/InsertTest.java | 6 -- .../operations/InsertUpdateIfConditionTest.java | 6 ++ .../cql3/validation/operations/UpdateTest.java | 69 +++- 10 files changed, 202 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 145afb9..e69bf08 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.11 + * Fix DELETE and UPDATE queries with empty IN restrictions (CASSANDRA-12829) * Mark MVs as built after successful bootstrap (CASSANDRA-12984) * Estimated TS drop-time histogram updated with Cell.NO_DELETION_TIME (CASSANDRA-13040) * Nodetool compactionstats fails with NullPointerException (CASSANDRA-13021) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java -- diff --git a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java index 647d22f..542dec9 100644 --- a/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java +++ b/src/java/org/apache/cassandra/cql3/restrictions/StatementRestrictions.java @@ -430,6 +430,17 @@ public final class StatementRestrictions } /** + * Checks if restrictions on the clustering key have IN restrictions. + * + * @return true if the restrictions on the clustering key have IN restrictions, + * false otherwise. + */ +public boolean clusteringKeyRestrictionsHasIN() +{ +return clusteringColumnsRestrictions.isIN(); +} + +/** * Processes the clustering column restrictions. * * @param hasQueriableIndex true if some of the queried data are indexed, false otherwise http://git-wip-us.apache.org/repos/asf/cassandra/blob/9fc1ffb6/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java -- diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java index f0aa835..76a6460 100644 --- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java @@ -401,7 +401,7 @@ public class BatchStatement implements CQLStatement QueryOptions statementOptions = options.forStatement(i); long timestamp = attrs.getTimestamp(now, statementOptions); List pks = statement.buildPartitionKeyNames(statementOptions); -if (pks.size() > 1) +if (statement.getRestrictions().keyIsInRelation()) throw new IllegalArgumentException("Batch with conditions cannot span multiple partitions (you cannot use IN on the partition key)"); if (key == null) { @@ -413,12 +413,11 @@ public class BatchStatement implements CQLStatement throw new InvalidRequestException("Batch with conditions cannot span multiple partitions"); } -SortedSet clusterings = statement.createClustering(statementOptions); + checkFalse(statement.getRestrictions().clusteringKeyRestrictionsHasIN(), + "IN on the clustering key columns is not supported with conditional %s", + statement.type.isUpdate()? "updates" : "deletions"); -
[jira] [Commented] (CASSANDRA-5051) Allow automatic cleanup after gc_grace
[ https://issues.apache.org/jira/browse/CASSANDRA-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751853#comment-15751853 ] Jon Haddad commented on CASSANDRA-5051: --- Maybe time for this in 4.0? > Allow automatic cleanup after gc_grace > -- > > Key: CASSANDRA-5051 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5051 > Project: Cassandra > Issue Type: New Feature >Reporter: Brandon Williams >Assignee: Vijay > Labels: vnodes > Fix For: 3.x > > Attachments: 0001-5051-v4.patch, 0001-5051-v6.patch, > 0001-5051-with-test-fixes.patch, 0001-CASSANDRA-5051.patch, > 0002-5051-remove-upgradesstable-v4.patch, > 0002-5051-remove-upgradesstable.patch, 0004-5051-additional-test-v4.patch, > 5051-v2.txt > > > When using vnodes, after adding a new node you have to run cleanup on all the > machines, because you don't know which are affected and chances are it was > most if not all of them. As an alternative to this intensive process, we > could allow cleanup during compaction if the data is older than gc_grace (or > perhaps some other time period since people tend to use gc_grace hacks to get > rid of tombstones.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12829) DELETE query with an empty IN clause can delete more than expected
[ https://issues.apache.org/jira/browse/CASSANDRA-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751847#comment-15751847 ] Benjamin Lerer commented on CASSANDRA-12829: +1 > DELETE query with an empty IN clause can delete more than expected > -- > > Key: CASSANDRA-12829 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12829 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Arch Linux x64, kernel 4.7.6, Cassandra 3.9 downloaded > from the website >Reporter: Jason T. Bradshaw >Assignee: Alex Petrov > > When deleting from a table with a certain structure and using an *in* clause > with an empty list, the *in* clause with an empty list can be ignored, > resulting in deleting more than is expected. > *Setup:* > {code} > cqlsh> create table test (a text, b text, id uuid, primary key ((a, b), id)); > cqlsh> insert into test (a, b, id) values ('a', 'b', > ----); > cqlsh> insert into test (a, b, id) values ('b', 'c', > ----); > cqlsh> insert into test (a, b, id) values ('a', 'c', > ----); > cqlsh> select * from test; > a | b | id > ---+---+-- > a | c | ---- > b | c | ---- > a | b | ---- > (3 rows) > {code} > *Expected:* > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in (); > cqlsh> select * from test; > a | b | id > ---+---+-- > a | c | ---- > b | c | ---- > a | b | ---- > (3 rows) > {code} > *Actual:* > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c') and id in (); > cqlsh> select * from test; > a | b | id > ---+---+-- > b | c | ---- > (1 rows) > {code} > Instead of deleting nothing, as the final empty *in* clause would imply, it > instead deletes everything that matches the first two clauses, acting as if > the following query had been issued instead: > {code} > cqlsh> delete from test where a = 'a' and b in ('a', 'b', 'c'); > {code} > This seems to be related to the presence of a tuple clustering key, as I > could not reproduce it without one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12859) Column-level permissions
[ https://issues.apache.org/jira/browse/CASSANDRA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751746#comment-15751746 ] Boris Melamed commented on CASSANDRA-12859: --- Thank you Stefan. Since that public commit in my fork, I've written more edge case dtests. Some of them discovered a bug which I've fixed (when accessing columns, some of which are granted on different roles of the user). To be published on my fork. I'm currently adding more tests around MVs. Is there a spec for what's the expected behavior with permissions and MVs? There are not many dtests for those, and from reading the code, it seems quite different from RDBMS view permissions. For example, in C*: - No permissions are required on an MV for SELECT. Only its base table must be granted. - For modifying a base table, the user must have permissions on all its MVs. This is starkly different from RDBMS, IMHO. I can try and proceed likewise for columns, but would like to make sure this is expected. > Column-level permissions > > > Key: CASSANDRA-12859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12859 > Project: Cassandra > Issue Type: New Feature > Components: Core, CQL >Reporter: Boris Melamed > Labels: doc-impacting > Attachments: Cassandra Proposal - Column-level permissions v2.docx, > Cassandra Proposal - Column-level permissions.docx > > Original Estimate: 504h > Remaining Estimate: 504h > > h4. Here is a draft of: > Cassandra Proposal - Column-level permissions.docx (attached) > h4. Quoting the 'Overview' section: > The purpose of this proposal is to add column-level (field-level) permissions > to Cassandra. It is my intent to soon start implementing this feature in a > fork, and to submit a pull request once it’s ready. > h4. Motivation > Cassandra already supports permissions on keyspace and table (column family) > level. Sources: > * http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra > * https://cassandra.apache.org/doc/latest/cql/security.html#data-control > At IBM, we have use cases in the area of big data analytics where > column-level access permissions are also a requirement. All industry RDBMS > products are supporting this level of permission control, and regulators are > expecting it from all data-based systems. > h4. Main day-one requirements > # Extend CQL (Cassandra Query Language) to be able to optionally specify a > list of individual columns, in the {{GRANT}} statement. The relevant > permission types are: {{MODIFY}} (for {{UPDATE}} and {{INSERT}}) and > {{SELECT}}. > # Persist the optional information in the appropriate system table > ‘system_auth.role_permissions’. > # Enforce the column access restrictions during execution. Details: > #* Should fit with the existing permission propagation down a role chain. > #* Proposed message format when a user’s roles give access to the queried > table but not to all of the selected, inserted, or updated columns: > "User %s has no %s permission on column %s of table %s" > #* Error will report only the first checked column. > Nice to have: list all inaccessible columns. > #* Error code is the same as for table access denial: 2100. > h4. Additional day-one requirements > # Reflect the column-level permissions in statements of type > {{LIST ALL PERMISSIONS OF someuser;}} > # When columns are dropped or renamed, trigger purging or adapting of their > permissions > # Performance should not degrade in any significant way. > # Backwards compatibility > #* Permission enforcement for DBs created before the upgrade should continue > to work with the same behavior after upgrading to a version that allows > column-level permissions. > #* Previous CQL syntax will remain valid, and have the same effect as before. > h4. Documentation > * > https://cassandra.apache.org/doc/latest/cql/security.html#grammar-token-permission > * Feedback request: any others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-13047) Point cqlsh help to the new doc
Sylvain Lebresne created CASSANDRA-13047: Summary: Point cqlsh help to the new doc Key: CASSANDRA-13047 URL: https://issues.apache.org/jira/browse/CASSANDRA-13047 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Fix For: 4.0 Cqlsh still points to the "old" textile CQL doc, but that's not really maintain anymore now that we have the new doc (which include everything the old doc had and more). We should update cqlsh to point to the new doc so we can remove the old one completely. Any takers? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11678) cassandra crush when enable hints_compression
[ https://issues.apache.org/jira/browse/CASSANDRA-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749713#comment-15749713 ] Daniel Miranda edited comment on CASSANDRA-11678 at 12/15/16 3:44 PM: -- I am observing this issue in 3.9.0 without any external crashes or reboots being involved. I'm running a 3-node test cluster in AWS. I have KairosDB [1] writing synthetic data from a stress program into the cluster. After a few minutes of stress one or more nodes crash with corrupted hint files. After the initial crash, I wiped one of the nodes clean and rebootstrapped, and in the next attempt the other two nodes crashed. I have hints_compression set to LZ4, and I can send a copy of the logs and corrupted hint files if they are of interest. [1]: http://kairosdb.github.io was (Author: danielkza): I am observing this issue in 3.9.0 without any external crashes or reboots being involved. I'm running a 3-node test cluster in AWS. I have KairosDB [1] writing synthetic data from a stress program into the cluster. Affter a few minutes of stress one or more nodes crash with corrupted hint files. After the initial crash, I wiped one of the nodes clean and rebootstrapped, and in the next attempt the other two nodes crashed. I have hints_compression set to LZ4, and I can send a copy of the logs and corrupted hint files if they are of interest. > cassandra crush when enable hints_compression > - > > Key: CASSANDRA-11678 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11678 > Project: Cassandra > Issue Type: Bug > Components: Core, Local Write-Read Paths > Environment: Centos 7 >Reporter: Weijian Lin >Assignee: Blake Eggleston >Priority: Critical > > When I enable hints_compression and set the compression class to > LZ4Compressor,the > cassandra (v3.05, V3.5.0) will crush。That is a bug, or any conf is wrong? > *Exception in V 3.5.0 * > {code} > ERROR [HintsDispatcher:2] 2016-04-26 15:02:56,970 > HintsDispatchExecutor.java:225 - Failed to dispatch hints file > abc4dda2-b551-427e-bb0b-e383d4a392e1-1461654138963-1.hints: file is > corrupted ({}) > org.apache.cassandra.io.FSReadError: java.io.EOFException > at > org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259) > [apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242) > [apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220) > [apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199) > [apache-cassandra-3.5.0.jar:3.5.0] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65] > Caused by: java.io.EOFException: null > at > org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:146) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.io.util.RebufferingInputStream.readPrimitiveSlowly(RebufferingInputStream.java:108) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.io.util.RebufferingInputStream.readInt(RebufferingInputStream.java:188) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNextInternal(HintsReader.java:297) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at >
[jira] [Commented] (CASSANDRA-12859) Column-level permissions
[ https://issues.apache.org/jira/browse/CASSANDRA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751678#comment-15751678 ] Sylvain Lebresne commented on CASSANDRA-12859: -- bq. There's {{doc/cql3/CQL.textile}} ({{ant generate-cql-html}}) Don't bother, this is basically deprecated and should really have gone away already. bq. and {{doc/source/cql/security.rst}} that could use some hints how this works {{ant gen-doc}} > Column-level permissions > > > Key: CASSANDRA-12859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12859 > Project: Cassandra > Issue Type: New Feature > Components: Core, CQL >Reporter: Boris Melamed > Labels: doc-impacting > Attachments: Cassandra Proposal - Column-level permissions v2.docx, > Cassandra Proposal - Column-level permissions.docx > > Original Estimate: 504h > Remaining Estimate: 504h > > h4. Here is a draft of: > Cassandra Proposal - Column-level permissions.docx (attached) > h4. Quoting the 'Overview' section: > The purpose of this proposal is to add column-level (field-level) permissions > to Cassandra. It is my intent to soon start implementing this feature in a > fork, and to submit a pull request once it’s ready. > h4. Motivation > Cassandra already supports permissions on keyspace and table (column family) > level. Sources: > * http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra > * https://cassandra.apache.org/doc/latest/cql/security.html#data-control > At IBM, we have use cases in the area of big data analytics where > column-level access permissions are also a requirement. All industry RDBMS > products are supporting this level of permission control, and regulators are > expecting it from all data-based systems. > h4. Main day-one requirements > # Extend CQL (Cassandra Query Language) to be able to optionally specify a > list of individual columns, in the {{GRANT}} statement. The relevant > permission types are: {{MODIFY}} (for {{UPDATE}} and {{INSERT}}) and > {{SELECT}}. > # Persist the optional information in the appropriate system table > ‘system_auth.role_permissions’. > # Enforce the column access restrictions during execution. Details: > #* Should fit with the existing permission propagation down a role chain. > #* Proposed message format when a user’s roles give access to the queried > table but not to all of the selected, inserted, or updated columns: > "User %s has no %s permission on column %s of table %s" > #* Error will report only the first checked column. > Nice to have: list all inaccessible columns. > #* Error code is the same as for table access denial: 2100. > h4. Additional day-one requirements > # Reflect the column-level permissions in statements of type > {{LIST ALL PERMISSIONS OF someuser;}} > # When columns are dropped or renamed, trigger purging or adapting of their > permissions > # Performance should not degrade in any significant way. > # Backwards compatibility > #* Permission enforcement for DBs created before the upgrade should continue > to work with the same behavior after upgrading to a version that allows > column-level permissions. > #* Previous CQL syntax will remain valid, and have the same effect as before. > h4. Documentation > * > https://cassandra.apache.org/doc/latest/cql/security.html#grammar-token-permission > * Feedback request: any others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12859) Column-level permissions
[ https://issues.apache.org/jira/browse/CASSANDRA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751659#comment-15751659 ] Stefan Podkowinski commented on CASSANDRA-12859: This looks really promising. Just played around with your branch and it worked as expected for me. Can you also update the documentation for syntax changes and examples? There's {{doc/cql3/CQL.textile}} ({{ant generate-cql-html}}) and {{doc/source/cql/security.rst}} that could use some hints how this works. > Column-level permissions > > > Key: CASSANDRA-12859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12859 > Project: Cassandra > Issue Type: New Feature > Components: Core, CQL >Reporter: Boris Melamed > Labels: doc-impacting > Attachments: Cassandra Proposal - Column-level permissions v2.docx, > Cassandra Proposal - Column-level permissions.docx > > Original Estimate: 504h > Remaining Estimate: 504h > > h4. Here is a draft of: > Cassandra Proposal - Column-level permissions.docx (attached) > h4. Quoting the 'Overview' section: > The purpose of this proposal is to add column-level (field-level) permissions > to Cassandra. It is my intent to soon start implementing this feature in a > fork, and to submit a pull request once it’s ready. > h4. Motivation > Cassandra already supports permissions on keyspace and table (column family) > level. Sources: > * http://www.datastax.com/dev/blog/role-based-access-control-in-cassandra > * https://cassandra.apache.org/doc/latest/cql/security.html#data-control > At IBM, we have use cases in the area of big data analytics where > column-level access permissions are also a requirement. All industry RDBMS > products are supporting this level of permission control, and regulators are > expecting it from all data-based systems. > h4. Main day-one requirements > # Extend CQL (Cassandra Query Language) to be able to optionally specify a > list of individual columns, in the {{GRANT}} statement. The relevant > permission types are: {{MODIFY}} (for {{UPDATE}} and {{INSERT}}) and > {{SELECT}}. > # Persist the optional information in the appropriate system table > ‘system_auth.role_permissions’. > # Enforce the column access restrictions during execution. Details: > #* Should fit with the existing permission propagation down a role chain. > #* Proposed message format when a user’s roles give access to the queried > table but not to all of the selected, inserted, or updated columns: > "User %s has no %s permission on column %s of table %s" > #* Error will report only the first checked column. > Nice to have: list all inaccessible columns. > #* Error code is the same as for table access denial: 2100. > h4. Additional day-one requirements > # Reflect the column-level permissions in statements of type > {{LIST ALL PERMISSIONS OF someuser;}} > # When columns are dropped or renamed, trigger purging or adapting of their > permissions > # Performance should not degrade in any significant way. > # Backwards compatibility > #* Permission enforcement for DBs created before the upgrade should continue > to work with the same behavior after upgrading to a version that allows > column-level permissions. > #* Previous CQL syntax will remain valid, and have the same effect as before. > h4. Documentation > * > https://cassandra.apache.org/doc/latest/cql/security.html#grammar-token-permission > * Feedback request: any others? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11995) Commitlog replaced with all NULs
[ https://issues.apache.org/jira/browse/CASSANDRA-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644485#comment-15644485 ] John Marsten edited comment on CASSANDRA-11995 at 12/15/16 3:31 PM: We observed this phenomenon running Apache Cassandra 3.7 on Windows Server 2012 VM. The precise history and state of the VM in question was unclear, but apparently the administrators took a live volume snapshot, and after reverting to the snapshot, Cassandra would no longer start due to the error above. In this case, the commit log file was 48MB of NUL bytes. The Cassandra system log showed an ungraceful termination and a handful of warnings related to slow commit log syncs, but no errors prior to the CommitLogReplayException on subsequent startups. Cassandra was configured for batch mode commit log syncing with a window size of 50ms. It's not clear whether the system was writing to Cassandra at the time of the snapshot. was (Author: jmarsten): We observed this phenomenon running Apache Cassandra 3.8 on Windows Server 2012 VM. The precise history and state of the VM in question was unclear, but apparently the administrators took a live snapshot, and after reverting to the snapshot, Cassandra would no longer start due to the error above. In this case, the commit log file was 48MB of NUL bytes. The Cassandra system log showed an ungraceful termination and a handful of warnings related to slow commit log syncs, but no errors prior to the CommitLogReplayException on subsequent startups. Cassandra was configured for batch mode commit log syncing with a window size of 50ms. It's not clear whether the system was writing to Cassandra at the time of the snapshot. > Commitlog replaced with all NULs > > > Key: CASSANDRA-11995 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11995 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Windows 10 Enterprise 1511 > DataStax Cassandra Community Server 2.2.3 >Reporter: James Howe > > I noticed this morning that Cassandra was failing to start, after being shut > down on Friday. > {code} > ERROR 09:13:37 Exiting due to error while processing commit log during > initialization. > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Could not read commit log descriptor in file C:\Program Files\DataStax > Community\data\commitlog\CommitLog-5-1465571056722.log > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:302) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:273) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:513) > [apache-cassandra-2.2.3.jar:2.2.3] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) > [apache-cassandra-2.2.3.jar:2.2.3] > {code} > Checking the referenced file reveals it comprises 33,554,432 (32 * 1024 * > 1024) NUL bytes. > No logs (stdout, stderr, prunsrv) from the shutdown show any other issues and > appear exactly as normal. > Is installed as a service via DataStax's distribution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-13046) Missing PID file directory when installing from RPM
Marcel Dopita created CASSANDRA-13046: - Summary: Missing PID file directory when installing from RPM Key: CASSANDRA-13046 URL: https://issues.apache.org/jira/browse/CASSANDRA-13046 Project: Cassandra Issue Type: Bug Components: Packaging Environment: CentOS 7.3 Reporter: Marcel Dopita Priority: Minor As already reported in CASSANDRA-10920, changes to /var/run/cassandra folder are not persistent across reboots and therefore saving pid file to that location fails. This causes trouble when managing the status of the cassandra service. The linked issue mentions good enough (and working) workaround (compared to writing new service file for systemd). Trunk now contains files necessary to produce RHEL compatible RPM in the /redhat folder. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x
[ https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751349#comment-15751349 ] Sylvain Lebresne commented on CASSANDRA-12620: -- Thanks, +1 assuming the tests are clean (nit: I don't think you need the {{Row.Deletion}} import anymore). > Resurrected empty rows on update to 3.x > --- > > Key: CASSANDRA-12620 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12620 > Project: Cassandra > Issue Type: Bug >Reporter: Collin Sauve >Assignee: Benjamin Lerer > > We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 > according to documentation), and were entering TTLs at write-time using the > DataStax C# Driver (using the POCO mapper). > Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that: > * should have been TTL'd > * have no non-primary-key column data > {code} > CREATE TABLE applicationservices.aggregate_bucket_event_v3 ( > bucket_type int, > bucket_id text, > date timestamp, > aggregate_id text, > event_type int, > event_id text, > entities list>>, > identity_sid text, > PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, > event_id) > ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, > event_id ASC) > AND bloom_filter_fp_chance = 0.1 > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = '' > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} > AND compression = {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = '99PERCENTILE'; > {code} > {code} > { > "partition" : { > "key" : [ "0", "26492" ], > "position" : 54397932 > }, > "rows" : [ > { > "type" : "row", > "position" : 54397961, > "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", > "773665449947099136" ], > "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : > 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false }, > "cells" : [ > { "name" : "identity_sid", "value" : "p_tw_zahidana" }, > { "name" : "entities", "deletion_info" : { "marked_deleted" : > "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } > }, > { "name" : "entities", "path" : [ > "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : > "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" }, > { "name" : "entities", "path" : [ > "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" } > ] > }, > { > "type" : "row", >}, > { > "type" : "row", > "position" : 54397177, > "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", > "765850666296225792" ], > "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397227, > "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", > "765805367347601409" ], > "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397276, > "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", > "765760069858365441" ], > "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" }, > "cells" : [ ] > }, > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x
[ https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751327#comment-15751327 ] Benjamin Lerer commented on CASSANDRA-12620: Thanks for the review. I updated my [branch|https://github.com/apache/cassandra/compare/trunk...blerer:12620-3.0?expand=1] with your change and tested it. It work perfectly including the usecase that was not working with my original fix. I will run the upgrade tests and post the results once I got them. > Resurrected empty rows on update to 3.x > --- > > Key: CASSANDRA-12620 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12620 > Project: Cassandra > Issue Type: Bug >Reporter: Collin Sauve >Assignee: Benjamin Lerer > > We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 > according to documentation), and were entering TTLs at write-time using the > DataStax C# Driver (using the POCO mapper). > Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that: > * should have been TTL'd > * have no non-primary-key column data > {code} > CREATE TABLE applicationservices.aggregate_bucket_event_v3 ( > bucket_type int, > bucket_id text, > date timestamp, > aggregate_id text, > event_type int, > event_id text, > entities list>>, > identity_sid text, > PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, > event_id) > ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, > event_id ASC) > AND bloom_filter_fp_chance = 0.1 > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = '' > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} > AND compression = {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = '99PERCENTILE'; > {code} > {code} > { > "partition" : { > "key" : [ "0", "26492" ], > "position" : 54397932 > }, > "rows" : [ > { > "type" : "row", > "position" : 54397961, > "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", > "773665449947099136" ], > "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : > 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false }, > "cells" : [ > { "name" : "identity_sid", "value" : "p_tw_zahidana" }, > { "name" : "entities", "deletion_info" : { "marked_deleted" : > "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } > }, > { "name" : "entities", "path" : [ > "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : > "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" }, > { "name" : "entities", "path" : [ > "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" } > ] > }, > { > "type" : "row", >}, > { > "type" : "row", > "position" : 54397177, > "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", > "765850666296225792" ], > "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397227, > "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", > "765805367347601409" ], > "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397276, > "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", > "765760069858365441" ], > "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" }, > "cells" : [ ] > }, > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12620) Resurrected empty rows on update to 3.x
[ https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751089#comment-15751089 ] Sylvain Lebresne edited comment on CASSANDRA-12620 at 12/15/16 12:47 PM: - Not sure this is the proper fix (as in, it's likely fine for anyone that isn't doing wacky things, but it doesn't stricly respect backward compatibility). In 2.x, I believe we never delete a row marker explicitely, so it can only be a tombstone due to TTLs. But a row marker expiring does not absolutely guarantee the whole row is gone. Consider the following example (which is weird, for sure, but possible): {noformat} CREATE TABLE t (k int PRIMARY KEY, v1 int, v2 int); UPDATE t USING TIMESTAMP 0 SET v1 = 1 WHERE k = 0; INSERT INTO t(k, v2) VALUES (0, 2) USING TIMESTAMP 1 AND TTL 3; {noformat} When the 2nd insert expires, we'll still have a row with {{v1=1}} (though no row marker internally), and this both in 2.x and 3.x. With this patch however, we might end up generating a row deletion with timestamp 1, which _would_ delete the first UPDATE, which is stricly speaking incorrect (for the CQL semantic). Long story short, a deleted row marker is not equivalent to a row deletion, it just mean there is no row marker, and as the equivalent in 3.x of a row marker is the primary key liveness info, all we want here is make sure we don't have one. In other words, I believe the proper patch is: {noformat} assert !cell.value.hasRemaining(); - builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, cell.ttl, cell.localDeletionTime)); + if (!cell.isTombstone()) + builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, cell.ttl, cell.localDeletionTime)); {noformat} was (Author: slebresne): Not sure this is the proper fix (as in, it's likely fine for anyone that isn't doing wacky things, but it doesn't stricly respect backward compatibility). In 2.x, I believe we never delete a row marker explicitely, so it can only be a tombstone due to TTLs. But a row marker expiring does not absolutely guarantee the whole row is gone. Consider the following example (which is weird, for sure, but possible): {noformat} CREATE TABLE t (k int PRIMARY KEY, v1 int, v2 int); UPDATE t USING TIMESTAMP 0 SET v1 = 1 WHERE k = 0; INSERT INTO t(k, v2) VALUES (0, 2) USING TIMESTAMP 1 AND TTL 3; {noformat} When the 2nd insert expires, we'll still have a row with {{v1=1}} (though no row marker internally), and this both in 2.x and 3.x. With this patch however, we might end up generating a row deletion with timestamp 1, which _would_ delete the first UPDATE, which is stricly speaking incorrect (for the CQL semantic). Long story short, a deleted row marker is not equivalent to a row deletion, it just mean there is no row marker, and as the equivalent in 3.x of a row marker is the primary key liveness info, all we want here is make sure we don't have one. In other words, I believe the proper patch is: {noformat} assert !cell.value.hasRemaining(); - builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, cell.ttl, cell.localDeletionTime)); + if (cell.isTombstone()) + builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, cell.ttl, cell.localDeletionTime)); {noformat} > Resurrected empty rows on update to 3.x > --- > > Key: CASSANDRA-12620 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12620 > Project: Cassandra > Issue Type: Bug >Reporter: Collin Sauve >Assignee: Benjamin Lerer > > We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 > according to documentation), and were entering TTLs at write-time using the > DataStax C# Driver (using the POCO mapper). > Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that: > * should have been TTL'd > * have no non-primary-key column data > {code} > CREATE TABLE applicationservices.aggregate_bucket_event_v3 ( > bucket_type int, > bucket_id text, > date timestamp, > aggregate_id text, > event_type int, > event_id text, > entities list>>, > identity_sid text, > PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, > event_id) > ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, > event_id ASC) > AND bloom_filter_fp_chance = 0.1 > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = '' > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} > AND compression = {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 >
[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x
[ https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751089#comment-15751089 ] Sylvain Lebresne commented on CASSANDRA-12620: -- Not sure this is the proper fix (as in, it's likely fine for anyone that isn't doing wacky things, but it doesn't stricly respect backward compatibility). In 2.x, I believe we never delete a row marker explicitely, so it can only be a tombstone due to TTLs. But a row marker expiring does not absolutely guarantee the whole row is gone. Consider the following example (which is weird, for sure, but possible): {noformat} CREATE TABLE t (k int PRIMARY KEY, v1 int, v2 int); UPDATE t USING TIMESTAMP 0 SET v1 = 1 WHERE k = 0; INSERT INTO t(k, v2) VALUES (0, 2) USING TIMESTAMP 1 AND TTL 3; {noformat} When the 2nd insert expires, we'll still have a row with {{v1=1}} (though no row marker internally), and this both in 2.x and 3.x. With this patch however, we might end up generating a row deletion with timestamp 1, which _would_ delete the first UPDATE, which is stricly speaking incorrect (for the CQL semantic). Long story short, a deleted row marker is not equivalent to a row deletion, it just mean there is no row marker, and as the equivalent in 3.x of a row marker is the primary key liveness info, all we want here is make sure we don't have one. In other words, I believe the proper patch is: {noformat} assert !cell.value.hasRemaining(); - builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, cell.ttl, cell.localDeletionTime)); + if (cell.isTombstone()) + builder.addPrimaryKeyLivenessInfo(LivenessInfo.create(cell.timestamp, cell.ttl, cell.localDeletionTime)); {noformat} > Resurrected empty rows on update to 3.x > --- > > Key: CASSANDRA-12620 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12620 > Project: Cassandra > Issue Type: Bug >Reporter: Collin Sauve >Assignee: Benjamin Lerer > > We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 > according to documentation), and were entering TTLs at write-time using the > DataStax C# Driver (using the POCO mapper). > Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that: > * should have been TTL'd > * have no non-primary-key column data > {code} > CREATE TABLE applicationservices.aggregate_bucket_event_v3 ( > bucket_type int, > bucket_id text, > date timestamp, > aggregate_id text, > event_type int, > event_id text, > entities list>>, > identity_sid text, > PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, > event_id) > ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, > event_id ASC) > AND bloom_filter_fp_chance = 0.1 > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = '' > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} > AND compression = {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = '99PERCENTILE'; > {code} > {code} > { > "partition" : { > "key" : [ "0", "26492" ], > "position" : 54397932 > }, > "rows" : [ > { > "type" : "row", > "position" : 54397961, > "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", > "773665449947099136" ], > "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : > 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false }, > "cells" : [ > { "name" : "identity_sid", "value" : "p_tw_zahidana" }, > { "name" : "entities", "deletion_info" : { "marked_deleted" : > "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } > }, > { "name" : "entities", "path" : [ > "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : > "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" }, > { "name" : "entities", "path" : [ > "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" } > ] > }, > { > "type" : "row", >}, > { > "type" : "row", > "position" : 54397177, > "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", > "765850666296225792" ], > "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397227, >
[jira] [Updated] (CASSANDRA-12620) Resurrected empty rows on update to 3.x
[ https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-12620: --- Reviewer: Sylvain Lebresne Status: Patch Available (was: In Progress) > Resurrected empty rows on update to 3.x > --- > > Key: CASSANDRA-12620 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12620 > Project: Cassandra > Issue Type: Bug >Reporter: Collin Sauve >Assignee: Benjamin Lerer > > We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 > according to documentation), and were entering TTLs at write-time using the > DataStax C# Driver (using the POCO mapper). > Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that: > * should have been TTL'd > * have no non-primary-key column data > {code} > CREATE TABLE applicationservices.aggregate_bucket_event_v3 ( > bucket_type int, > bucket_id text, > date timestamp, > aggregate_id text, > event_type int, > event_id text, > entities list>>, > identity_sid text, > PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, > event_id) > ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, > event_id ASC) > AND bloom_filter_fp_chance = 0.1 > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = '' > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} > AND compression = {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = '99PERCENTILE'; > {code} > {code} > { > "partition" : { > "key" : [ "0", "26492" ], > "position" : 54397932 > }, > "rows" : [ > { > "type" : "row", > "position" : 54397961, > "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", > "773665449947099136" ], > "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : > 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false }, > "cells" : [ > { "name" : "identity_sid", "value" : "p_tw_zahidana" }, > { "name" : "entities", "deletion_info" : { "marked_deleted" : > "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } > }, > { "name" : "entities", "path" : [ > "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : > "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" }, > { "name" : "entities", "path" : [ > "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" } > ] > }, > { > "type" : "row", >}, > { > "type" : "row", > "position" : 54397177, > "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", > "765850666296225792" ], > "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397227, > "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", > "765805367347601409" ], > "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397276, > "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", > "765760069858365441" ], > "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" }, > "cells" : [ ] > }, > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12620) Resurrected empty rows on update to 3.x
[ https://issues.apache.org/jira/browse/CASSANDRA-12620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751054#comment-15751054 ] Benjamin Lerer commented on CASSANDRA-12620: Sorry, for the delay. It tooks me some time and a certain amount of back and forth between the {{2.1}} and {{3.0}} code base to fully understand the problem. [~bric3] Thanks for the SSTables they helped a lot. In {{2.1}} when a row is created with a {{TTL}} the row marker cell is created as an {{ExpiringCell}} but when the row is compacted if the {{TTL}} has expired the cell will be converted into a {{DeletedCell}}. In {{3.0}} the code used to read the post 3.0 format was only expecting an {{ExpiringCell}} for rows with {{TTL}} and was ignoring the local deletion time of the {{DeletedCell}}. Due to that the row was not marked as deleted anymore. As the problem was in the code used to deserialize the post 3.0 format it was not necessary to run {{upgradesstables}} to see the problem. I originally could not reproduce the problem because I did not think of running a compaction before upgrading (SizeTieredCompaction). I pushed a patch [here|https://github.com/apache/cassandra/compare/trunk...blerer:12620-3.0?expand=1]. > Resurrected empty rows on update to 3.x > --- > > Key: CASSANDRA-12620 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12620 > Project: Cassandra > Issue Type: Bug >Reporter: Collin Sauve >Assignee: Benjamin Lerer > > We had the below table on C* 2.x (dse 4.8.4, we assume was 2.1.15.1423 > according to documentation), and were entering TTLs at write-time using the > DataStax C# Driver (using the POCO mapper). > Upon upgrade to 3.0.8.1293 (DSE 5.0.2), we are seeing a lot of rows that: > * should have been TTL'd > * have no non-primary-key column data > {code} > CREATE TABLE applicationservices.aggregate_bucket_event_v3 ( > bucket_type int, > bucket_id text, > date timestamp, > aggregate_id text, > event_type int, > event_id text, > entities list>>, > identity_sid text, > PRIMARY KEY ((bucket_type, bucket_id), date, aggregate_id, event_type, > event_id) > ) WITH CLUSTERING ORDER BY (date DESC, aggregate_id ASC, event_type ASC, > event_id ASC) > AND bloom_filter_fp_chance = 0.1 > AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} > AND comment = '' > AND compaction = {'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'} > AND compression = {'chunk_length_in_kb': '64', 'class': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND crc_check_chance = 1.0 > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = '99PERCENTILE'; > {code} > {code} > { > "partition" : { > "key" : [ "0", "26492" ], > "position" : 54397932 > }, > "rows" : [ > { > "type" : "row", > "position" : 54397961, > "clustering" : [ "2016-09-07 23:33Z", "3651664", "0", > "773665449947099136" ], > "liveness_info" : { "tstamp" : "2016-09-07T23:34:09.758Z", "ttl" : > 172741, "expires_at" : "2016-09-09T23:33:10Z", "expired" : false }, > "cells" : [ > { "name" : "identity_sid", "value" : "p_tw_zahidana" }, > { "name" : "entities", "deletion_info" : { "marked_deleted" : > "2016-09-07T23:34:09.757999Z", "local_delete_time" : "2016-09-07T23:34:09Z" } > }, > { "name" : "entities", "path" : [ > "936e17e1-7553-11e6-9b92-29a33b5827c3" ], "value" : > "0:https\\://www.youtube.com/watch?v=pwAJAssv6As" }, > { "name" : "entities", "path" : [ > "936e17e2-7553-11e6-9b92-29a33b5827c3" ], "value" : "2:youtube" } > ] > }, > { > "type" : "row", >}, > { > "type" : "row", > "position" : 54397177, > "clustering" : [ "2016-08-17 10:00Z", "6387376", "0", > "765850666296225792" ], > "liveness_info" : { "tstamp" : "2016-08-17T11:26:15.917001Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397227, > "clustering" : [ "2016-08-17 07:00Z", "6387376", "0", > "765805367347601409" ], > "liveness_info" : { "tstamp" : "2016-08-17T08:11:17.587Z" }, > "cells" : [ ] > }, > { > "type" : "row", > "position" : 54397276, > "clustering" : [ "2016-08-17 04:00Z", "6387376", "0", > "765760069858365441" ], > "liveness_info" : { "tstamp" : "2016-08-17T05:58:11.228Z" }, > "cells" : [ ] > }, > {code} -- This message was
[jira] [Commented] (CASSANDRA-12990) More fixes to the TokenAllocator
[ https://issues.apache.org/jira/browse/CASSANDRA-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751003#comment-15751003 ] Branimir Lambov commented on CASSANDRA-12990: - Since we don't get the rack info, the new adapter should not [offer a means of getting it|https://github.com/DikangGu/cassandra/commit/d986e89b7b21b5464bd1de7908ee5cb1d12dfd30#diff-bf046c281d96767f4c3c4fe97eeaf938R220] at all. That method shouldn't be called by the no-replication allocator, let's make that clear there and then by adding a comment and throwing. The adapter should also just return 1 for the replica count, so that the [0 check in {{TokenAllocatorFactory}}|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/dht/tokenallocator/TokenAllocatorFactory.java#L37] can be removed and the 0 replica hack is only done in one place in the code. Now that {{waitForGossipToSettle}} is in {{Gossiper}}, the name is somewhat redundant. Rename to {{waitToSettle}}? > More fixes to the TokenAllocator > > > Key: CASSANDRA-12990 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12990 > Project: Cassandra > Issue Type: Bug >Reporter: Dikang Gu >Assignee: Dikang Gu > Fix For: 3.12 > > > CASSANDRA-12983 fixes the NoReplicationTokenAllocator in the replication > factor equals 0 case. > We find one more problem that, in our big cluster, ~1000 nodes, the Gossip > takes longer time to settle down, so the TokenAllocator may throw NPE when > trying to find the dc rack information from the Gossip metadata, in the > getStrategy function. > This patch will fix it, it does two things: > 1. wait gossip to settle down before trying to allocating tokens. > 2. for replication factor equals 0 or 1 case, we do not need to check the > topology for rack information, since each node will be treated separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12905) Retry acquire MV lock on failure instead of throwing WTE on streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750958#comment-15750958 ] Benjamin Roth edited comment on CASSANDRA-12905 at 12/15/16 10:00 AM: -- +1 for the dtest! Thanks for explaining that hint-thing. For my understanding of hint retransmission: If we have a hint file that is being processed and a WTE occurs in the middle, will the whole file be retransmitted or can it be resumed at the last successful position? I guess this is not the case from my personal observations. I had situations with > 1GB hint queues per sender node which were not going to shrink due to WTEs. It seemed like the same hints have been retransmitted over and over again from scratch instead of trying to resume. What helped in this situation was to pause hint delivery on all nodes but 1-2. I'm pretty sure the problem was WTEs due to lock contentions. To be honest, I did not try lowering hinted_handoff_throttle_in_kb or at least I don't remember. was (Author: brstgt): +1 for the dtest! For my understanding of hint retransmission: If we have a hint file that is being processed and a WTE occurs in the middle, will the whole file be retransmitted or can it be resumed at the last successful position? I guess this is not the case from my personal observations. I had situations with > 1GB hint queues per sender node which were not going to shrink due to WTEs. It seemed like the same hints have been retransmitted over and over again from scratch instead of trying to resume. What helped in this situation was to pause hint delivery on all nodes but 1-2. I'm pretty sure the problem was WTEs due to lock contentions. To be honest, I did not try lowering hinted_handoff_throttle_in_kb or at least I don't remember. > Retry acquire MV lock on failure instead of throwing WTE on streaming > - > > Key: CASSANDRA-12905 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12905 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: centos 6.7 x86_64 >Reporter: Nir Zilka >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.10 > > > Hello, > I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, > private VLAN), > first it was 2.2.5.1 and repair worked flawlessly, > second upgrade was to 3.0.9 (with upgradesstables) and also repair worked > well, > then i upgraded 2 weeks ago to 3.9 - and the repair problems started. > there are several errors types from the system.log (different nodes) : > - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx > - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation > timed out - received only 0 responses > - Remote peer xxx.xxx.xxx.xxx failed stream session > - Session completed with the following error > org.apache.cassandra.streaming.StreamException: Stream failed > > i use 3.9 default configuration with the cluster settings adjustments (3 > seeds, GossipingPropertyFileSnitch). > streaming_socket_timeout_in_ms is the default (8640). > i'm afraid from consistency problems while i'm not performing repair. > Any ideas? > Thanks, > Nir. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12905) Retry acquire MV lock on failure instead of throwing WTE on streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750958#comment-15750958 ] Benjamin Roth commented on CASSANDRA-12905: --- +1 for the dtest! For my understanding of hint retransmission: If we have a hint file that is being processed and a WTE occurs in the middle, will the whole file be retransmitted or can it be resumed at the last successful position? I guess this is not the case from my personal observations. I had situations with > 1GB hint queues per sender node which were not going to shrink due to WTEs. It seemed like the same hints have been retransmitted over and over again from scratch instead of trying to resume. What helped in this situation was to pause hint delivery on all nodes but 1-2. I'm pretty sure the problem was WTEs due to lock contentions. To be honest, I did not try lowering hinted_handoff_throttle_in_kb or at least I don't remember. > Retry acquire MV lock on failure instead of throwing WTE on streaming > - > > Key: CASSANDRA-12905 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12905 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: centos 6.7 x86_64 >Reporter: Nir Zilka >Assignee: Benjamin Roth >Priority: Critical > Fix For: 3.10 > > > Hello, > I performed two upgrades to the current cluster (currently 15 nodes, 1 DC, > private VLAN), > first it was 2.2.5.1 and repair worked flawlessly, > second upgrade was to 3.0.9 (with upgradesstables) and also repair worked > well, > then i upgraded 2 weeks ago to 3.9 - and the repair problems started. > there are several errors types from the system.log (different nodes) : > - Sync failed between /xxx.xxx.xxx.xxx and /xxx.xxx.xxx.xxx > - Streaming error occurred on session with peer xxx.xxx.xxx.xxx Operation > timed out - received only 0 responses > - Remote peer xxx.xxx.xxx.xxx failed stream session > - Session completed with the following error > org.apache.cassandra.streaming.StreamException: Stream failed > > i use 3.9 default configuration with the cluster settings adjustments (3 > seeds, GossipingPropertyFileSnitch). > streaming_socket_timeout_in_ms is the default (8640). > i'm afraid from consistency problems while i'm not performing repair. > Any ideas? > Thanks, > Nir. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908 ] Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:43 AM: - I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} * Calculates the likeliness of a race condition leading to unnecessary repairs * @see https://issues.apache.org/jira/browse/CASSANDRA-12991 * * This assumes that all writes are equally spread over all token ranges * and there is one subrange repair executed for each existing token range 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 2.93% Unnecessary range repairs per repair: 60.00 8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 5.37% Unnecessary range repairs per repair: 110.00 {quote} You may ask why I entered latencies like 10ms or 20ms - this seems quite high. It is indeed quite high for regular tables and a cluster that is not overloaded. Under these conditions, the latency is dominated by your network latency, so 1ms seems quite fair to me. As soon as you use MVs and your cluster tends to overload, higher latencies are not unrealistic. You have to take into account that an MV operation does read before write and the latency may vary very much. For MVs the latency is not (only) any more dominated by network latency but by MV lock aquisition and read before write. Both factors can introduce MUCH higher latencies, depending on concurrent operations on MV, number of SSTables, compaction strategy, just everything that affects read performance. If your cluster is overloaded, these effects have an even higher impact. I observed MANY situations on our production system where writes timed out during streaming because of lock contention and or RBW impacts. These situations mainly pop up during repair sessions when streams cause bulk mutation applies (see StreamReceiverTask path for MVs). Impact is even higher due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the situation even more unpredictable and increases "drifts" of nodes, like Node A is overloaded but Node B not because Node A receives a stream from a different repair but Node B does not. This is a vicious circle driven several factors: - Stream puts pressure on nodes - especially larg(er) partitions - hints tend to queue up - hint delivery puts more pressure - retransmission of failed hint delivery puts even more pressure - latencies go up - stream validations drift - more (unnecessary) streams - goto 0 This calculation example is just hypothetic. This *may* happen as calculated but it totally depends on the model, cluster dimensions, cluster load, write activity, distribution of writes and repair execution. I don't claim that fixing this issue will remove all MV performance problems but it may be helps to remove one impediment in the mentioned vicious circle. My proposal is NOT to control flushes. This is far too complicated and wont help. A flush, whenever it may happen and whatever range it flushes may or may not contain a mutation that _should_ be there. What helps is to cut off all data retrospectively at a synchronized and fix timestamp when executing the validation. You can define a grace period (GP). When you start validation at VS on the repair coordinator, then you expect all mutations to arrive no later than VS that were created before VS - GP. That can be done at SSTable scanner level by filtering all events (cells, tombstones) after VS - GP during validation compaction. Something like the opposite of purging tombstones after GCGS. was (Author: brstgt): I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} * Calculates the likeliness of a race condition leading to unnecessary repairs * @see https://issues.apache.org/jira/browse/CASSANDRA-12991 * * This assumes that all writes are equally spread over all token ranges * and there is one subrange repair executed for each existing token range 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation
[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908 ] Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:34 AM: - I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} * Calculates the likeliness of a race condition leading to unnecessary repairs * @see https://issues.apache.org/jira/browse/CASSANDRA-12991 * * This assumes that all writes are equally spread over all token ranges * and there is one subrange repair executed for each existing token range 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 2.93% Unnecessary range repairs per repair: 60.00 8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 5.37% Unnecessary range repairs per repair: 110.00 {quote} You may ask why I entered latencies like 10ms or 20ms - this seems quite high. It is indeed quite high for regular tables and a cluster that is not overloaded. Under these conditions, the latency is dominated by your network latency, so 1ms seems quite fair to me. As soon as you use MVs and your cluster tends to overload, higher latencies are not unrealistic. You have to take into account that an MV operation does read before write and the latency may vary very much. For MVs the latency is not (only) any more dominated by network latency but by MV lock aquisition and read before write. Both factors can introduce MUCH higher latencies, depending on concurrent operations on MV, number of SSTables, compaction strategy, just everything that affects read performance. If your cluster is overloaded, these effects have an even higher impact. I observed MANY situations on our production system where writes timed out during streaming because of lock contention and or RBW impacts. These situations mainly pop up during repair sessions when streams cause bulk mutation applies (see StreamReceiverTask path for MVs). Impact is even higher due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the situation even more unpredictable and increases "drifts" of nodes, like Node A is overloaded but Node B not because Node A receives a stream from a different repair but Node B does not. This is a vicious circle driven several factors: - Stream puts pressure on nodes - especially larg(er) partitions - hints tend to queue up - hint delivery puts more pressure - retransmission of failed hint delivery puts even more pressure - latencies go up - stream validations drift - more (unnecessary) streams - goto 0 This calculation example is just hypothetic. This *may* happen as calculated but it totally depends on the model, cluster dimensions, cluster load, write activity, distribution of writes and repair execution. I don't claim that fixing this issue will remove all MV performance problems but it may be helps to remove one impediment in the mentioned vicious circle. My proposal is NOT to control flushes. This is far too complicated and wont help. A flush, whenever it may happen and whatever range it flushes may or may not contain a mutation that _should_ be there. The only thing that helps is to cut off all data retrospectively at a synchronized and fix timestamp when executing the validation. You can only define a grace period (GP). When you start validation at VS on the repair coordinator, then you expect all mutations to arrive no later than VS that were created before VS - GP. That can IMHO only be done at SSTable scanner level by filtering all events (cells, tombstones) after VS - GP during validation compaction. Something like the opposite of purging tombstones after GCGS. was (Author: brstgt): I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for
[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908 ] Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:34 AM: - I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 2.93% Unnecessary range repairs per repair: 60.00 8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 5.37% Unnecessary range repairs per repair: 110.00 {quote} You may ask why I entered latencies like 10ms or 20ms - this seems quite high. It is indeed quite high for regular tables and a cluster that is not overloaded. Under these conditions, the latency is dominated by your network latency, so 1ms seems quite fair to me. As soon as you use MVs and your cluster tends to overload, higher latencies are not unrealistic. You have to take into account that an MV operation does read before write and the latency may vary very much. For MVs the latency is not (only) any more dominated by network latency but by MV lock aquisition and read before write. Both factors can introduce MUCH higher latencies, depending on concurrent operations on MV, number of SSTables, compaction strategy, just everything that affects read performance. If your cluster is overloaded, these effects have an even higher impact. I observed MANY situations on our production system where writes timed out during streaming because of lock contention and or RBW impacts. These situations mainly pop up during repair sessions when streams cause bulk mutation applies (see StreamReceiverTask path for MVs). Impact is even higher due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the situation even more unpredictable and increases "drifts" of nodes, like Node A is overloaded but Node B not because Node A receives a stream from a different repair but Node B does not. This is a vicious circle driven several factors: - Stream puts pressure on nodes - especially larg(er) partitions - hints tend to queue up - hint delivery puts more pressure - retransmission of failed hint delivery puts even more pressure - latencies go up - stream validations drift - more (unnecessary) streams - goto 0 This calculation example is just hypothetic. This *may* happen as calculated but it totally depends on the model, cluster dimensions, cluster load, write activity, distribution of writes and repair execution. I don't claim that fixing this issue will remove all MV performance problems but it may be helps to remove one impediment in the mentioned vicious circle. My proposal is NOT to control flushes. This is far too complicated and wont help. A flush, whenever it may happen and whatever range it flushes may or may not contain a mutation that _should_ be there. The only thing that helps is to cut off all data retrospectively at a synchronized and fix timestamp when executing the validation. You can only define a grace period (GP). When you start validation at VS on the repair coordinator, then you expect all mutations to arrive no later than VS that were created before VS - GP. That can IMHO only be done at SSTable scanner level by filtering all events (cells, tombstones) after VS - GP during validation compaction. Something like the opposite of purging tombstones after GCGS. was (Author: brstgt): I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 2.93% Unnecessary range repairs per repair: 60.00 8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 5.37% Unnecessary range repairs per repair: 110.00 {/quote} You may ask why I entered
[jira] [Comment Edited] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908 ] Benjamin Roth edited comment on CASSANDRA-12991 at 12/15/16 9:33 AM: - I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 2.93% Unnecessary range repairs per repair: 60.00 8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 5.37% Unnecessary range repairs per repair: 110.00 {/quote} You may ask why I entered latencies like 10ms or 20ms - this seems quite high. It is indeed quite high for regular tables and a cluster that is not overloaded. Under these conditions, the latency is dominated by your network latency, so 1ms seems quite fair to me. As soon as you use MVs and your cluster tends to overload, higher latencies are not unrealistic. You have to take into account that an MV operation does read before write and the latency may vary very much. For MVs the latency is not (only) any more dominated by network latency but by MV lock aquisition and read before write. Both factors can introduce MUCH higher latencies, depending on concurrent operations on MV, number of SSTables, compaction strategy, just everything that affects read performance. If your cluster is overloaded, these effects have an even higher impact. I observed MANY situations on our production system where writes timed out during streaming because of lock contention and or RBW impacts. These situations mainly pop up during repair sessions when streams cause bulk mutation applies (see StreamReceiverTask path for MVs). Impact is even higher due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the situation even more unpredictable and increases "drifts" of nodes, like Node A is overloaded but Node B not because Node A receives a stream from a different repair but Node B does not. This is a vicious circle driven several factors: - Stream puts pressure on nodes - especially larg(er) partitions - hints tend to queue up - hint delivery puts more pressure - retransmission of failed hint delivery puts even more pressure - latencies go up - stream validations drift - more (unnecessary) streams - goto 0 This calculation example is just hypothetic. This *may* happen as calculated but it totally depends on the model, cluster dimensions, cluster load, write activity, distribution of writes and repair execution. I don't claim that fixing this issue will remove all MV performance problems but it may be helps to remove one impediment in the mentioned vicious circle. My proposal is NOT to control flushes. This is far too complicated and wont help. A flush, whenever it may happen and whatever range it flushes may or may not contain a mutation that _should_ be there. The only thing that helps is to cut off all data retrospectively at a synchronized and fix timestamp when executing the validation. You can only define a grace period (GP). When you start validation at VS on the repair coordinator, then you expect all mutations to arrive no later than VS that were created before VS - GP. That can IMHO only be done at SSTable scanner level by filtering all events (cells, tombstones) after VS - GP during validation compaction. Something like the opposite of purging tombstones after GCGS. was (Author: brstgt): I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} * Calculates the likeliness of a race condition leading to unnecessary repairs * @see https://issues.apache.org/jira/browse/CASSANDRA-12991 * * This assumes that all writes are equally spread over all token ranges * and there is one subrange repair executed for each existing token range 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness
[jira] [Commented] (CASSANDRA-12991) Inter-node race condition in validation compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-12991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750908#comment-15750908 ] Benjamin Roth commented on CASSANDRA-12991: --- I created a little script to calculate some possible scenarios: https://gist.github.com/brstgt/447533208f6afa25a77c9a963b114f99 Output: {quote} * Calculates the likeliness of a race condition leading to unnecessary repairs * @see https://issues.apache.org/jira/browse/CASSANDRA-12991 * * This assumes that all writes are equally spread over all token ranges * and there is one subrange repair executed for each existing token range 3 Nodes, 256 Tokens / Node, 1ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 0.39% Unnecessary range repairs per repair: 3.00 3 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 1000 req/s Total Ranges: 768 Likeliness for RC per range: 1.56% Unnecessary range repairs per repair: 12.00 8 Nodes, 256 Tokens / Node, 10ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 2.93% Unnecessary range repairs per repair: 60.00 8 Nodes, 256 Tokens / Node, 20ms Mutation Latency, 1ms Validation Latency, 5000 req/s Total Ranges: 2048 Likeliness for RC per range: 5.37% Unnecessary range repairs per repair: 110.00 {/quote} You may ask why I entered latencies like 10ms or 20ms - this seems quite high. It is indeed quite high for regular tables and a cluster that is not overloaded. Under these conditions, the latency is dominated by your network latency, so 1ms seems quite fair to me. As soon as you use MVs and your cluster tends to overload, higher latencies are not unrealistic. You have to take into account that an MV operation does read before write and the latency may vary very much. For MVs the latency is not (only) any more dominated by network latency but by MV lock aquisition and read before write. Both factors can introduce MUCH higher latencies, depending on concurrent operations on MV, number of SSTables, compaction strategy, just everything that affects read performance. If your cluster is overloaded, these effects have an even higher impact. I observed MANY situations on our production system where writes timed out during streaming because of lock contention and or RBW impacts. These situations mainly pop up during repair sessions when streams cause bulk mutation applies (see StreamReceiverTask path for MVs). Impact is even higher due to CASSANDRA-12888. Parallel repairs like e.g. reaper does, makes the situation even more unpredictable and increases "drifts" of nodes, like Node A is overloaded but Node B not because Node A receives a stream from a different repair but Node B does not. This is a vicious circle driven several factors: - Stream puts pressure on nodes - especially larg(er) partitions - hints tend to queue up - hint delivery puts more pressure - retransmission of failed hint delivery puts even more pressure - latencies go up - stream validations drift - more (unnecessary) streams - goto 0 This calculation example is just hypothetic. This *may* happen as calculated but it totally depends on the model, cluster dimensions, cluster load, write activity, distribution of writes and repair execution. I don't claim that fixing this issue will remove all MV performance problems but it may be helps to remove one impediment in the mentioned vicious circle. My proposal is NOT to control flushes. This is far too complicated and wont help. A flush, whenever it may happen and whatever range it flushes may or may not contain a mutation that _should_ be there. The only thing that helps is to cut off all data retrospectively at a synchronized and fix timestamp when executing the validation. You can only define a grace period (GP). When you start validation at VS on the repair coordinator, then you expect all mutations to arrive no later than VS that were created before VS - GP. That can IMHO only be done at SSTable scanner level by filtering all events (cells, tombstones) after VS - GP during validation compaction. Something like the opposite of purging tombstones after GCGS. > Inter-node race condition in validation compaction > -- > > Key: CASSANDRA-12991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12991 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Roth >Priority: Minor > > Problem: > When a validation compaction is triggered by a repair it may happen that due > to flying in mutations the merkle trees differ but the data is consistent > however. > Example: > t = 1: > Repair starts, triggers validations > Node A starts validation > t = 10001: > Mutation arrives at Node A > t = 10002: > Mutation arrives at
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750847#comment-15750847 ] Nitsan Wakart commented on CASSANDRA-13006: --- OnOutOfMemoryError and HeapDumpOnOutOfMemoryError are both available on Zing. The default for HeapDumpOnOutOfMemoryError is false. Let me know if you need more details. > Disable automatic heap dumps on OOM error > - > > Key: CASSANDRA-13006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13006 > Project: Cassandra > Issue Type: Bug > Components: Configuration >Reporter: anmols >Assignee: Benjamin Lerer >Priority: Minor > Fix For: 3.0.9 > > Attachments: 13006-3.0.9.txt > > > With CASSANDRA-9861, a change was added to enable collecting heap dumps by > default if the process encountered an OOM error. These heap dumps are stored > in the Apache Cassandra home directory unless configured otherwise (see > [Cassandra Support > Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] > for this feature). > > The creation and storage of heap dumps aides debugging and investigative > workflows, but is not be desirable for a production environment where these > heap dumps may occupy a large amount of disk space and require manual > intervention for cleanups. > > Managing heap dumps on out of memory errors and configuring the paths for > these heap dumps are available as JVM options in JVM. The current behavior > conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. > > A patch can be proposed here that would make the heap dump on OOM error honor > the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate > heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM > option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8496) Remove MemtablePostFlusher
[ https://issues.apache.org/jira/browse/CASSANDRA-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750807#comment-15750807 ] Branimir Lambov commented on CASSANDRA-8496: Bumping priority as we have seen production issues made worse by incorrect behaviour of forced flush and snapshot after a failed flush. > Remove MemtablePostFlusher > -- > > Key: CASSANDRA-8496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8496 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Branimir Lambov > > To improve clearing of the CL, prevent infinite growth, and ensure the prompt > completion of tasks waiting on flush in the case of transient errors, large > flushes or slow disks, in 2.1 we could eliminate the post flusher altogether. > Since we now enforce that Memtables track contiguous ranges, a relatively > small change would permit Memtables to know the exact minimum as well as the > currently known exact maximum. The CL could easily track the total dirty > range, knowing that it must be contiguous, by using an AtomicLong instead of > an AtomicInteger, and tracking both the min/max seen, not just the max. The > only slight complexity will come in for tracking the _clean_ range as this > can now be non-contiguous, if there are 3 memtable flushes covering the same > CL segment, and one of them completes later. To solve this we can use an > interval tree since these operations are infrequent, so the extra overhead is > nominal. Once the interval tree completely overlaps the dirty range, we mark > the entire dirty range clean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8496) Remove MemtablePostFlusher
[ https://issues.apache.org/jira/browse/CASSANDRA-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Branimir Lambov updated CASSANDRA-8496: --- Priority: Major (was: Minor) > Remove MemtablePostFlusher > -- > > Key: CASSANDRA-8496 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8496 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Branimir Lambov > > To improve clearing of the CL, prevent infinite growth, and ensure the prompt > completion of tasks waiting on flush in the case of transient errors, large > flushes or slow disks, in 2.1 we could eliminate the post flusher altogether. > Since we now enforce that Memtables track contiguous ranges, a relatively > small change would permit Memtables to know the exact minimum as well as the > currently known exact maximum. The CL could easily track the total dirty > range, knowing that it must be contiguous, by using an AtomicLong instead of > an AtomicInteger, and tracking both the min/max seen, not just the max. The > only slight complexity will come in for tracking the _clean_ range as this > can now be non-contiguous, if there are 3 memtable flushes covering the same > CL segment, and one of them completes later. To solve this we can use an > interval tree since these operations are infrequent, so the extra overhead is > nominal. Once the interval tree completely overlaps the dirty range, we mark > the entire dirty range clean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)