[jira] [Commented] (CASSANDRA-7281) SELECT on tuple relations are broken for mixed ASC/DESC clustering order
[ https://issues.apache.org/jira/browse/CASSANDRA-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568783#comment-14568783 ] Benjamin Lerer commented on CASSANDRA-7281: --- Sorry, my comment was really not that clear. In 2.1: I think that your code should tackle the problem within the {{SelectStatement.buildBound}} method. The tests for that method are within {{SelectStatementTest}}. In 2.2 you are right the code should be within {{MultiColumnRestriction.Slice.appendBoundTo}} method and the tests are within {{PrimaryKeyRestrictionSetTest }}. I expect that if you manage to fix the problem in 2.1 as part of the {{SelectStatement.buildBound}} method you should be able to make it work within 2.2 without too much difficulties by extracting your fix and putting it within {{MultiColumnRestriction.Slice.appendBoundTo}}. Hope it helps. Do not hesitate to contact me on IRC or by email if you have more questions SELECT on tuple relations are broken for mixed ASC/DESC clustering order Key: CASSANDRA-7281 URL: https://issues.apache.org/jira/browse/CASSANDRA-7281 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Marcin Szymaniuk Fix For: 2.1.x Attachments: 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v2.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v3.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v4.patch As noted on [CASSANDRA-6875|https://issues.apache.org/jira/browse/CASSANDRA-6875?focusedCommentId=13992153page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13992153], the tuple notation is broken when the clustering order mixes ASC and DESC directives because the range of data they describe don't correspond to a single continuous slice internally. To copy the example from CASSANDRA-6875: {noformat} cqlsh:ks create table foo (a int, b int, c int, PRIMARY KEY (a, b, c)) WITH CLUSTERING ORDER BY (b DESC, c ASC); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 2, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 1); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 0, 0); cqlsh:ks SELECT * FROM foo WHERE a=0; a | b | c ---+---+--- 0 | 2 | 0 0 | 1 | 0 0 | 1 | 1 0 | 0 | 0 (4 rows) cqlsh:ks SELECT * FROM foo WHERE a=0 AND (b, c) (1, 0); a | b | c ---+---+--- 0 | 2 | 0 (1 rows) {noformat} The last query should really return {{(0, 2, 0)}} and {{(0, 1, 1)}}. For that specific example we should generate 2 internal slices, but I believe that with more clustering columns we may have more slices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568835#comment-14568835 ] Stefania commented on CASSANDRA-8656: - Thanks [~mshuler], with a loop it is indeed easy to reproduce and if fails on average 2 or 3 times every 10 iterations. I cherry picked the commit for CASSANDRA-8739 and that seems to have fixed it. I've run the loop twice and I got no failures. See also the attached branch, a 2.0 with CASSANDRA-8739, hopefully it will start running on Jenkins soon. [~krummas] do you agree with the analysis and if so can you merge CASSANRA-8739 to 2.0? Here is the bash code if someone wants to test this further: {code} #!/bin/bash for i in `seq 1 10`; do echo echo Run iteration $i ant long-test -Dtest.name=LongLeveledCompactionStrategyTest done {code} long-test LongLeveledCompactionStrategyTest flaps in 2.0 Key: CASSANDRA-8656 URL: https://issues.apache.org/jira/browse/CASSANDRA-8656 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Priority: Minor Labels: test-failure Fix For: 2.0.x Attachments: system.log LongLeveledCompactionStrategyTest periodically fails with: {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.412 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97) [junit] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409) [junit] Caused by: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) [junit] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) [junit] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [junit] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) [junit] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87) [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED {noformat} I would guess the failure is
[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table
[ https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568708#comment-14568708 ] Piotr Kołaczkowski commented on CASSANDRA-7688: --- So what is this JMX call to refresh the estimates? I tried setting {{-Dcassandra.size_recorder_interval=1}} (I know, ridiculously low value) and the size estimates table is still empty when running the tests. Add data sizing to a system table - Key: CASSANDRA-7688 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688 Project: Cassandra Issue Type: New Feature Reporter: Jeremiah Jordan Assignee: Aleksey Yeschenko Fix For: 2.1.5 Attachments: 7688.txt Currently you can't implement something similar to describe_splits_ex purely from the a native protocol driver. https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily getting ownership information to a client in the java-driver. But you still need the data sizing part to get splits of a given size. We should add the sizing information to a system table so that native clients can get to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568806#comment-14568806 ] Sylvain Lebresne commented on CASSANDRA-9486: - bq. we switched maxOrderingSet to a LinkedList Make sense, I've pushed the update. I'll wait for the CI to run on it and commit, unless you have remarks on that last versions. LazilyCompactedRow accumulates all expired RangeTombstones -- Key: CASSANDRA-9486 URL: https://issues.apache.org/jira/browse/CASSANDRA-9486 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Assignee: Sylvain Lebresne Priority: Critical Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x Attachments: 0001-9486.patch LazilyCompactedRow initializes a ColumnIndex.Builder to use its RangeTombstone.Tracker, but it only calls update() with a RT argument, never an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So all the RT ever seen for the partition (that have expired) remain in memory until the compaction completes. To make matters worse, this then forces a linear scan of all of these RT for each live cell we add, so this extra load hangs around for a long time, and compactions stall. This issue is biting one of our users badly (at least, it seems likely to be this issue), and there may be others. This user is not even making use of RT extensively themselves, only collections (presumably with a complete overwrite of the contents of the collection, resulting in a RT being generated). Probably the best solution is to make the RT addition itself remove any already present that are no longer helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568827#comment-14568827 ] Benedict commented on CASSANDRA-9486: - LGTM. There's a double in (i.e. in in) in one of your comment modifications. Also, {{insertBefore}} doesn't need to call {{next()}} for the behaviour here, but not at all fussed since the semantics of an arbitrary method labelled {{insertBefore}} might well want to do so. LazilyCompactedRow accumulates all expired RangeTombstones -- Key: CASSANDRA-9486 URL: https://issues.apache.org/jira/browse/CASSANDRA-9486 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Assignee: Sylvain Lebresne Priority: Critical Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x Attachments: 0001-9486.patch LazilyCompactedRow initializes a ColumnIndex.Builder to use its RangeTombstone.Tracker, but it only calls update() with a RT argument, never an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So all the RT ever seen for the partition (that have expired) remain in memory until the compaction completes. To make matters worse, this then forces a linear scan of all of these RT for each live cell we add, so this extra load hangs around for a long time, and compactions stall. This issue is biting one of our users badly (at least, it seems likely to be this issue), and there may be others. This user is not even making use of RT extensively themselves, only collections (presumably with a complete overwrite of the contents of the collection, resulting in a RT being generated). Probably the best solution is to make the RT addition itself remove any already present that are no longer helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8609) Remove depency of hadoop to internals (Cell/CellName)
[ https://issues.apache.org/jira/browse/CASSANDRA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-8609: --- Summary: Remove depency of hadoop to internals (Cell/CellName) (was: t) Remove depency of hadoop to internals (Cell/CellName) - Key: CASSANDRA-8609 URL: https://issues.apache.org/jira/browse/CASSANDRA-8609 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Sam Tunnicliffe Fix For: 2.2.0 rc1 Attachments: 8609-2.2-2.txt, 8609-2.2.txt, CASSANDRA-8609-3.0-branch.txt For some reason most of the Hadoop code (ColumnFamilyRecordReader, CqlStorage, ...) uses the {{Cell}} and {{CellName}} classes. That dependency is entirely artificial: all this code is really client code that communicate with Cassandra over thrift/native protocol and there is thus no reason for it to use internal classes. And in fact, thoses classes are used in a very crude way, as a {{PairByteBuffer, ByteBuffer}} really. But this dependency is really painful when we make changes to the internals. Further, every time we do so, I believe we break some of those the APIs due to the change. This has been painful for CASSANDRA-5417 and this is now painful for CASSANDRA-8099. But while I somewhat hack over it in CASSANDRA-5417, this was a mistake and we should have removed the depency back then. So let do that now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7281) SELECT on tuple relations are broken for mixed ASC/DESC clustering order
[ https://issues.apache.org/jira/browse/CASSANDRA-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568757#comment-14568757 ] Marcin Szymaniuk commented on CASSANDRA-7281: - In 2.2 I cannot see classes/methods mentioned in your comment. From quick look it seems to me that: - MultiColumnRestriction.Slice#appendBoundTo is the place to implement the logic - PrimaryKeyRestrictionSetTest is good place to create more of unit-style tests. Can you please have a look and confirm? SELECT on tuple relations are broken for mixed ASC/DESC clustering order Key: CASSANDRA-7281 URL: https://issues.apache.org/jira/browse/CASSANDRA-7281 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Marcin Szymaniuk Fix For: 2.1.x Attachments: 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v2.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v3.patch, 0001-CASSANDRA-7281-SELECT-on-tuple-relations-are-broken-v4.patch As noted on [CASSANDRA-6875|https://issues.apache.org/jira/browse/CASSANDRA-6875?focusedCommentId=13992153page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13992153], the tuple notation is broken when the clustering order mixes ASC and DESC directives because the range of data they describe don't correspond to a single continuous slice internally. To copy the example from CASSANDRA-6875: {noformat} cqlsh:ks create table foo (a int, b int, c int, PRIMARY KEY (a, b, c)) WITH CLUSTERING ORDER BY (b DESC, c ASC); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 2, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 0); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 1, 1); cqlsh:ks INSERT INTO foo (a, b, c) VALUES (0, 0, 0); cqlsh:ks SELECT * FROM foo WHERE a=0; a | b | c ---+---+--- 0 | 2 | 0 0 | 1 | 0 0 | 1 | 1 0 | 0 | 0 (4 rows) cqlsh:ks SELECT * FROM foo WHERE a=0 AND (b, c) (1, 0); a | b | c ---+---+--- 0 | 2 | 0 (1 rows) {noformat} The last query should really return {{(0, 2, 0)}} and {{(0, 1, 1)}}. For that specific example we should generate 2 internal slices, but I believe that with more clustering columns we may have more slices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8609) t
[ https://issues.apache.org/jira/browse/CASSANDRA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-8609: --- Summary: t (was: Remove depency of hadoop to internals (Cell/CellName)) t - Key: CASSANDRA-8609 URL: https://issues.apache.org/jira/browse/CASSANDRA-8609 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Sam Tunnicliffe Fix For: 2.2.0 rc1 Attachments: 8609-2.2-2.txt, 8609-2.2.txt, CASSANDRA-8609-3.0-branch.txt For some reason most of the Hadoop code (ColumnFamilyRecordReader, CqlStorage, ...) uses the {{Cell}} and {{CellName}} classes. That dependency is entirely artificial: all this code is really client code that communicate with Cassandra over thrift/native protocol and there is thus no reason for it to use internal classes. And in fact, thoses classes are used in a very crude way, as a {{PairByteBuffer, ByteBuffer}} really. But this dependency is really painful when we make changes to the internals. Further, every time we do so, I believe we break some of those the APIs due to the change. This has been painful for CASSANDRA-5417 and this is now painful for CASSANDRA-8099. But while I somewhat hack over it in CASSANDRA-5417, this was a mistake and we should have removed the depency back then. So let do that now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania reassigned CASSANDRA-8656: --- Assignee: Stefania long-test LongLeveledCompactionStrategyTest flaps in 2.0 Key: CASSANDRA-8656 URL: https://issues.apache.org/jira/browse/CASSANDRA-8656 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Priority: Minor Labels: test-failure Fix For: 2.0.x Attachments: system.log LongLeveledCompactionStrategyTest periodically fails with: {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.412 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97) [junit] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409) [junit] Caused by: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) [junit] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) [junit] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [junit] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) [junit] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87) [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED {noformat} I would guess the failure is 10-20% of the time, looping over the test repeatedly. On the 2.1 branch, the failure is different, so perhaps this could also be updated. {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.04 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy cannot be cast to org.apache.cassandra.db.compaction.LeveledCompactionStrategy [junit] java.lang.ClassCastException:
[jira] [Updated] (CASSANDRA-9529) Standardize quoting in Unix scripts
[ https://issues.apache.org/jira/browse/CASSANDRA-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-9529: Description: {{$CLASSPATH}} and {{$JAVA}} are quoted in some scripts and not quoted in other scripts. Since it's best practice to use quotes, we should update the scripts without quotes around these variables. (was: {{$CLASSPATH}} and {{$JAVA}} are quoted in some scripts and not quoted in other scripts. Since it's best practice to use quotes, we should update the scripts were the quotes around these variables are missing.) Standardize quoting in Unix scripts --- Key: CASSANDRA-9529 URL: https://issues.apache.org/jira/browse/CASSANDRA-9529 Project: Cassandra Issue Type: Improvement Environment: All Unix based systems. Reporter: Stefania Assignee: Stefania Priority: Minor Fix For: 3.x, 2.1.x, 2.2.x, 2.0.16 {{$CLASSPATH}} and {{$JAVA}} are quoted in some scripts and not quoted in other scripts. Since it's best practice to use quotes, we should update the scripts without quotes around these variables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9533) Make batch commitlog mode easier to tune
[ https://issues.apache.org/jira/browse/CASSANDRA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569489#comment-14569489 ] Jonathan Ellis commented on CASSANDRA-9533: --- It's pretty clear that the new behavior is worse for most users. I thought we did it that way because we don't have a queue of operations to peek into anymore, so it's difficult to provide the old behavior of stop sleeping when the queue is empty. Strictly speaking the new behavior matches the comment *less* well since the comment said {{it will wait *up to*}} the window, which is in fact what it did. The new behavior waits *at least* the window. Make batch commitlog mode easier to tune Key: CASSANDRA-9533 URL: https://issues.apache.org/jira/browse/CASSANDRA-9533 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Assignee: Benedict Fix For: 3.x As discussed in CASSANDRA-9504, 2.1 changed commitlog_sync_batch_window_in_ms from a maximum time to wait between fsync to the minimum time, so one must be very careful to keep it small enough that most writers aren't kept waiting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[ https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569605#comment-14569605 ] Antoine Levy-Lambert commented on CASSANDRA-5543: - do you have a stack trace of when the java command invoked by ant is failing ? maybe running ant -verbose could give more output. can you give a pointer to the ant build file you are running ? Ant issues when building gen-cql2-grammar - Key: CASSANDRA-5543 URL: https://issues.apache.org/jira/browse/CASSANDRA-5543 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.3 Reporter: Joaquin Casares Assignee: Dave Brosius Priority: Trivial Below are the commands and outputs that were returned. The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant realclean` then it works fine after a second pass. {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml clean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/classes [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java [delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr realclean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build BUILD SUCCESSFUL Total time: 0 seconds {CODE} {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml maven-ant-tasks-localrepo: maven-ant-tasks-download: [echo] Downloading Maven ANT Tasks... [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar [get] To: /home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar maven-ant-tasks-init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib maven-declare-dependencies: maven-ant-tasks-retrieve-build: [artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from repository central at http://repo1.maven.org/maven2 [artifact:dependencies] [INFO] Unable to find resource 'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2 (http://download.java.net/maven/2) [artifact:dependencies] Building ant file: /home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml [copy] Copying 45 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/jars [copy] Copying 35 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/sources init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java check-avro-generate: avro-interface-generate-internode: [echo] Generating Avro internode code... avro-generate: build-subprojects: check-gen-cli-grammar: gen-cli-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g check-gen-cql2-grammar: gen-cql2-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g ... [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as IDENT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as K_KEY using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as QMARK using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as FLOAT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as STRING_LITERAL using multiple alternatives: 1, 2 [java] As a result, alternative(s)
[jira] [Commented] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[ https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569629#comment-14569629 ] Ron Kuris commented on CASSANDRA-9536: -- There are a few possible fixes to this, each has their pros and cons: # Don't record values less than 1 second; these are probably retransmits. This will prevent the phi average from being dropped below 1, which means that all phi can do is make the system less sensitive. The disadvantage to this is that flakey networks will still not receive any phi bonuses. # Whenever a packet is received longer than MAX_INTERVAL_IN_NANOS, just record an entry with MAX_INTERVAL_IN_NANOS instead of just dropping it. # Do both of the above I am leaning toward both, since this means that flakier networks will raise the phi average closer to 2, giving them twice the resilience before being marked down. The failure detector becomes more sensitive when the network is flakey -- Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Priority: Minor I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The median actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8609) Remove depency of hadoop to internals (Cell/CellName)
[ https://issues.apache.org/jira/browse/CASSANDRA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569536#comment-14569536 ] Sam Tunnicliffe commented on CASSANDRA-8609: Not just yet I'm afraid, there are still some dependencies on internals like {{IPartitioner}}, {{AbstractType}} in determining splits. Ideally, the Hadoop integration code shouldn't have any dependencies beyond the cql driver and thrift (and client-utils). For that to happen though, we'll need to duplicate a few internal things to enable the split sizing done entirely client side along the lines of [SPARKC-94|https://datastax-oss.atlassian.net/browse/SPARKC-94]). Maybe that should be done as part of CASSANDRA-9353. There are also some dependencies in {{CqlBulkRecordWriter}} ({{CFMetaData}}, {{Config}}, {{DatabaseDescriptor}}) which can't easily be removed yet. Remove depency of hadoop to internals (Cell/CellName) - Key: CASSANDRA-8609 URL: https://issues.apache.org/jira/browse/CASSANDRA-8609 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Sam Tunnicliffe Fix For: 2.2.0 rc1 Attachments: 8609-2.2-2.txt, 8609-2.2.txt, CASSANDRA-8609-3.0-branch.txt For some reason most of the Hadoop code (ColumnFamilyRecordReader, CqlStorage, ...) uses the {{Cell}} and {{CellName}} classes. That dependency is entirely artificial: all this code is really client code that communicate with Cassandra over thrift/native protocol and there is thus no reason for it to use internal classes. And in fact, thoses classes are used in a very crude way, as a {{PairByteBuffer, ByteBuffer}} really. But this dependency is really painful when we make changes to the internals. Further, every time we do so, I believe we break some of those the APIs due to the change. This has been painful for CASSANDRA-5417 and this is now painful for CASSANDRA-8099. But while I somewhat hack over it in CASSANDRA-5417, this was a mistake and we should have removed the depency back then. So let do that now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9535) KeyCache code should handle assertion errors from corrupt keycache better
Philip Thompson created CASSANDRA-9535: -- Summary: KeyCache code should handle assertion errors from corrupt keycache better Key: CASSANDRA-9535 URL: https://issues.apache.org/jira/browse/CASSANDRA-9535 Project: Cassandra Issue Type: Bug Reporter: Philip Thompson Fix For: 3.x, 2.2.x While running 2.2.0-beta1, a user ran into the following exception from a corrupt keycache file: {code} ERROR [main] 2015-05-30 10:31:18,603 CassandraDaemon.java:571 - Exception encountered during startup java.lang.AssertionError: null at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:202) ~[main/:na] at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:190) ~[main/:na] at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:144) ~[main/:na] at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:446) ~[main/:na] at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:142) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:328) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431) ~[main/:na] at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:324) ~[main/:na] at org.apache.cassandra.db.Keyspace.init(Keyspace.java:275) ~[main/:na] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:117) ~[main/:na] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:94) ~[main/:na] at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:281) ~[main/:na] at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:128) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:451) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:558) [main/:na] {code} This is from the failed assert of {{columnsIndex.size() 1}} at {code} private IndexedEntry(long position, DeletionTime deletionTime, ListIndexHelper.IndexInfo columnsIndex) { super(position); assert deletionTime != null; assert columnsIndex != null columnsIndex.size() 1; this.deletionTime = deletionTime; this.columnsIndex = columnsIndex; } {code} Discussing with [~thobbs], he felt that C* could do a better job of handling this, even though the root issue was a corrupt file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9071) CQLSSTableWriter gives java.lang.AssertionError: Empty partition
[ https://issues.apache.org/jira/browse/CASSANDRA-9071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569561#comment-14569561 ] Benedict commented on CASSANDRA-9071: - Heh. Thanks for pointing that out! I've pushed a fix and unit test [here|https://github.com/belliottsmith/cassandra/tree/9051] CQLSSTableWriter gives java.lang.AssertionError: Empty partition Key: CASSANDRA-9071 URL: https://issues.apache.org/jira/browse/CASSANDRA-9071 Project: Cassandra Issue Type: Bug Components: Tools Environment: java 7 / 8 cassandra 2.1.3 snapshot build locally with last commit https://github.com/apache/cassandra/commit/6ee4b0989d9a3ae3e704918622024fa57fdf63e7 macos Yosemite 10.10.2 Reporter: Ajit Joglekar Assignee: Fabien Rousseau Fix For: 2.1.x Attachments: EmailWriter.java, Screen Shot 2015-04-15 at 11.14.40 PM.png, cassandra-2.1-9071.txt, data.csv.gz I am always getting the following error: Exception in thread main java.lang.AssertionError: Empty partition at org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:228) It happens at a certain point that seems to be repeatable. Only issue is I am converting 400 million records into multiple SSTables and creating small test is a challenge Last comment from Benedict looks relevant here https://issues.apache.org/jira/browse/CASSANDRA-8619 Is there a work around, quick fix, fix that I can try out locally? Thanks, -Ajit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9533) Make batch commitlog mode easier to tune
[ https://issues.apache.org/jira/browse/CASSANDRA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569535#comment-14569535 ] Benedict commented on CASSANDRA-9533: - bq. matches the comment *less* well You're being a bit selective in which parts you bold :) it will wait up to implies _it will wait_ - which it would not, at all. The reference to Postgres' behaviour also indicates it will actually wait that period (although with separate sibling requirements, and on a microsecond time horizon which is much more sensible). Further, our docs say To avoid syncing after every write, Cassandra groups the mutations into batches and *syncs every* {{commitlog_batch_window_in_ms.}} Not at least that often, but - implcitly - exactly that often, as we do now. bq. I thought we did it that way because we don't have a queue of operations to peek into anymore, so it's difficult to provide the old behavior of stop sleeping when the queue is empty. No, unfortunately the best I can find of the etymology of this change is some offline discussion between Brandon, Jeremiah and myself, which occurred 2-3 months after commit: {quote} Benedict Elliott Smith so, just figured out why that CL unit test @yukim found went from hero to zero in 2.1 Benedict Elliott Smith SHANP on IRC has found the old Batch CL code doesn't behave in the same way Benedict Elliott Smith the window only serves as a maximum for buffering Benedict Elliott Smith so if you get one record arrive, it will immediately sync Benedict Elliott Smith in 2.1 this changed. i'm not sure if I got the wrong end of the stick (and it seems everyone else who's been discussing it since, maybe), or if this is a mistake that's been present all along Benedict Elliott Smith but we should probably decide which behaviour we want to go with in 2.1 dr driftxI don't think it was a mistake, exactly, I've explained it lots of times in training that way (pre-2.1 behavior) Benedict Elliott Smith right Benedict Elliott Smith so question is: did I just misunderstand, or did somebody tell me to implement it this way? and ignoring the answer, do we want to restore the old behaviour? Benedict Elliott Smith To avoid syncing after every write, Cassandra groups the mutations into batches and syncs every {{commitlog_batch_window_in_ms. }} Benedict Elliott Smith anyway, i'm easy, and heading to bed. if nobody says anything i'll leave it how it is :-) ZJeremiah D Jordan As long as the window time is the max time between syncs and writes block until they sync. I think your code is fine. As that is how we have documented it working all along Benedict Elliott Smith @jd my version has it as the the exact time between syncs {quote} So it is not at all clear why it happened. My recollection was that we discussed it and decided to normalise on the doc behaviour, but I cannot find a reference to that, so it is possible I simply implemented it how the docs described it, and you reviewed it with the same lens. Either way, it was decided to let the change stand due to it matching the doc descriptions and the lack of further feedback. We can certainly wait for the queue to empty, by issuing Barriers on the OpOrder; we already issue one and wait for any active at that moment to complete, which is probably good enough. If we want to wait until _none_ can be active, we can just (on completion of the barrier), check if there are any now running and issue another if there are. Make batch commitlog mode easier to tune Key: CASSANDRA-9533 URL: https://issues.apache.org/jira/browse/CASSANDRA-9533 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Assignee: Benedict Fix For: 3.x As discussed in CASSANDRA-9504, 2.1 changed commitlog_sync_batch_window_in_ms from a maximum time to wait between fsync to the minimum time, so one must be very careful to keep it small enough that most writers aren't kept waiting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9533) Make batch commitlog mode easier to tune
[ https://issues.apache.org/jira/browse/CASSANDRA-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569552#comment-14569552 ] Jonathan Ellis commented on CASSANDRA-9533: --- bq. We can certainly wait for the queue to empty, by issuing Barriers on the OpOrder; we already issue one and wait for any active at that moment to complete, which is probably good enough. WFM. Make batch commitlog mode easier to tune Key: CASSANDRA-9533 URL: https://issues.apache.org/jira/browse/CASSANDRA-9533 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Assignee: Benedict Fix For: 3.x As discussed in CASSANDRA-9504, 2.1 changed commitlog_sync_batch_window_in_ms from a maximum time to wait between fsync to the minimum time, so one must be very careful to keep it small enough that most writers aren't kept waiting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9535) KeyCache code should handle assertion errors from corrupt keycache better
[ https://issues.apache.org/jira/browse/CASSANDRA-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-9535: --- Issue Type: Improvement (was: New Feature) KeyCache code should handle assertion errors from corrupt keycache better - Key: CASSANDRA-9535 URL: https://issues.apache.org/jira/browse/CASSANDRA-9535 Project: Cassandra Issue Type: Improvement Reporter: Philip Thompson Fix For: 3.x, 2.2.x While running 2.2.0-beta1, a user ran into the following exception from a corrupt keycache file: {code} ERROR [main] 2015-05-30 10:31:18,603 CassandraDaemon.java:571 - Exception encountered during startup java.lang.AssertionError: null at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:202) ~[main/:na] at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:190) ~[main/:na] at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:144) ~[main/:na] at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:446) ~[main/:na] at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:142) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:328) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431) ~[main/:na] at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:324) ~[main/:na] at org.apache.cassandra.db.Keyspace.init(Keyspace.java:275) ~[main/:na] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:117) ~[main/:na] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:94) ~[main/:na] at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:281) ~[main/:na] at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:128) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:451) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:558) [main/:na] {code} This is from the failed assert of {{columnsIndex.size() 1}} at {code} private IndexedEntry(long position, DeletionTime deletionTime, ListIndexHelper.IndexInfo columnsIndex) { super(position); assert deletionTime != null; assert columnsIndex != null columnsIndex.size() 1; this.deletionTime = deletionTime; this.columnsIndex = columnsIndex; } {code} Discussing with [~thobbs], he felt that C* could do a better job of handling this, even though the root issue was a corrupt file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9535) KeyCache code should handle assertion errors from corrupt keycache better
[ https://issues.apache.org/jira/browse/CASSANDRA-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-9535: --- Issue Type: New Feature (was: Bug) KeyCache code should handle assertion errors from corrupt keycache better - Key: CASSANDRA-9535 URL: https://issues.apache.org/jira/browse/CASSANDRA-9535 Project: Cassandra Issue Type: New Feature Reporter: Philip Thompson Fix For: 3.x, 2.2.x While running 2.2.0-beta1, a user ran into the following exception from a corrupt keycache file: {code} ERROR [main] 2015-05-30 10:31:18,603 CassandraDaemon.java:571 - Exception encountered during startup java.lang.AssertionError: null at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:202) ~[main/:na] at org.apache.cassandra.db.RowIndexEntry$IndexedEntry.init(RowIndexEntry.java:190) ~[main/:na] at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:144) ~[main/:na] at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:446) ~[main/:na] at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:142) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:328) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:455) ~[main/:na] at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431) ~[main/:na] at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:324) ~[main/:na] at org.apache.cassandra.db.Keyspace.init(Keyspace.java:275) ~[main/:na] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:117) ~[main/:na] at org.apache.cassandra.db.Keyspace.open(Keyspace.java:94) ~[main/:na] at org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:281) ~[main/:na] at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:128) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:451) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:558) [main/:na] {code} This is from the failed assert of {{columnsIndex.size() 1}} at {code} private IndexedEntry(long position, DeletionTime deletionTime, ListIndexHelper.IndexInfo columnsIndex) { super(position); assert deletionTime != null; assert columnsIndex != null columnsIndex.size() 1; this.deletionTime = deletionTime; this.columnsIndex = columnsIndex; } {code} Discussing with [~thobbs], he felt that C* could do a better job of handling this, even though the root issue was a corrupt file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[ https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9536: -- Priority: Minor (was: Major) The failure detector becomes more sensitive when the network is flakey -- Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Bug Components: Core Reporter: Ron Kuris Priority: Minor I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The median actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[ https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9536: -- Issue Type: Improvement (was: Bug) The failure detector becomes more sensitive when the network is flakey -- Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Priority: Minor I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The median actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
Ron Kuris created CASSANDRA-9536: Summary: The failure detector becomes more sensitive when the network is flakey Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Bug Components: Core Reporter: Ron Kuris I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The median actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[ https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569642#comment-14569642 ] Ron Kuris commented on CASSANDRA-9536: -- Some additional notes from my research on how this code works: FailureDetector's report() is called each time the gossiper receives a message from that node. The message should arrive every second (this is not tunable). The failure detector records the time since the last message, up to 1000 samples, which will take about 16 minutes, 40 seconds. It does not record the time if the message was sent more than 2 seconds ago. In the gossiper's status check, it calls FailureDetector's interpret() method. This calculates phi, which is the time since the last message, divided by the average of the collected samples. It then divides by a constant, PHI_FACTOR, which is about 0.434, then compares to phi_convict_threshold. If the adjusted phi is greater than the threshold, the node is marked down. If the average collected samples drops below 1, this increases phi. I ran several additional tests, including changing the packet dropping time to 0-1 seconds, which severely lowers the average and causes the node to flip flop UP/DOWN. I'm very open to helping to fix this, but I'm afraid it might require a different implementation of the FailureDetector. Perhaps I should make it pluggable, where we have a new implementation of IFailureDetector? I'm open to suggestions. The failure detector becomes more sensitive when the network is flakey -- Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Priority: Minor I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The median actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569645#comment-14569645 ] Benjamin Lerer commented on CASSANDRA-9527: --- [~thobbs] can you review? Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x Attachments: 9527.txt In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[ https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Kuris updated CASSANDRA-9536: - Description: I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The mean actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. was: I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The median actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. The failure detector becomes more sensitive when the network is flakey -- Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Priority: Minor I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The mean actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[ https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Lococo updated CASSANDRA-5543: --- Attachment: 5543-ant.log The ant build-file I'm using is the top-level build.xml, if I understand your questions correctly. The results of ant -verbose 5543-ant.log are attached. The stderr also contains a stacktrace: $ ant -verbose ant.log BUILD FAILED /home/ubuntu/cassandra/build.xml:222: Java returned: 1 at org.apache.tools.ant.taskdefs.Java.execute(Java.java:111) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) at org.apache.tools.ant.Task.perform(Task.java:348) at org.apache.tools.ant.Target.execute(Target.java:435) at org.apache.tools.ant.Target.performTasks(Target.java:456) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1393) at org.apache.tools.ant.Project.executeTarget(Project.java:1364) at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) at org.apache.tools.ant.Project.executeTargets(Project.java:1248) at org.apache.tools.ant.Main.runBuild(Main.java:851) at org.apache.tools.ant.Main.startAnt(Main.java:235) at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280) at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109) Total time: 20 seconds Ant issues when building gen-cql2-grammar - Key: CASSANDRA-5543 URL: https://issues.apache.org/jira/browse/CASSANDRA-5543 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.3 Reporter: Joaquin Casares Assignee: Dave Brosius Priority: Trivial Attachments: 5543-ant.log Below are the commands and outputs that were returned. The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant realclean` then it works fine after a second pass. {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml clean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/classes [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java [delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr realclean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build BUILD SUCCESSFUL Total time: 0 seconds {CODE} {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml maven-ant-tasks-localrepo: maven-ant-tasks-download: [echo] Downloading Maven ANT Tasks... [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar [get] To: /home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar maven-ant-tasks-init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib maven-declare-dependencies: maven-ant-tasks-retrieve-build: [artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from repository central at http://repo1.maven.org/maven2 [artifact:dependencies] [INFO] Unable to find resource 'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2 (http://download.java.net/maven/2) [artifact:dependencies] Building ant file: /home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml [copy] Copying 45 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/jars [copy] Copying 35 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/sources init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java check-avro-generate: avro-interface-generate-internode: [echo] Generating Avro internode code... avro-generate: build-subprojects: check-gen-cli-grammar: gen-cli-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g check-gen-cql2-grammar: gen-cql2-grammar: [echo] Building Grammar
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[ https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569749#comment-14569749 ] Mike Lococo commented on CASSANDRA-5543: I'm also happy to provide direct access to a test-box (a temporary aws box created solely to test this issue) where this is happening if that's useful. Reach out to me via email with your ssh public-key and I'll respond with the ip address if that sounds helpful. Ant issues when building gen-cql2-grammar - Key: CASSANDRA-5543 URL: https://issues.apache.org/jira/browse/CASSANDRA-5543 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.3 Reporter: Joaquin Casares Assignee: Dave Brosius Priority: Trivial Attachments: 5543-ant.log Below are the commands and outputs that were returned. The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant realclean` then it works fine after a second pass. {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml clean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/classes [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java [delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr realclean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build BUILD SUCCESSFUL Total time: 0 seconds {CODE} {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml maven-ant-tasks-localrepo: maven-ant-tasks-download: [echo] Downloading Maven ANT Tasks... [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar [get] To: /home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar maven-ant-tasks-init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib maven-declare-dependencies: maven-ant-tasks-retrieve-build: [artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from repository central at http://repo1.maven.org/maven2 [artifact:dependencies] [INFO] Unable to find resource 'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2 (http://download.java.net/maven/2) [artifact:dependencies] Building ant file: /home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml [copy] Copying 45 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/jars [copy] Copying 35 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/sources init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java check-avro-generate: avro-interface-generate-internode: [echo] Generating Avro internode code... avro-generate: build-subprojects: check-gen-cli-grammar: gen-cli-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g check-gen-cql2-grammar: gen-cql2-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g ... [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as IDENT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as K_KEY using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as QMARK using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as FLOAT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as
[jira] [Commented] (CASSANDRA-9534) Parent repair data not always saved to system_distributed
[ https://issues.apache.org/jira/browse/CASSANDRA-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569762#comment-14569762 ] Jim Witschey commented on CASSANDRA-9534: - If [this dtest failure|http://cassci.datastax.com/view/trunk/job/trunk_dtest/200/testReport/repair_test/TestRepairDataSystemTable/repair_table_test/] is any indication, the regular repair history table is also not populated sometimes. Parent repair data not always saved to system_distributed - Key: CASSANDRA-9534 URL: https://issues.apache.org/jira/browse/CASSANDRA-9534 Project: Cassandra Issue Type: Bug Reporter: Jim Witschey Assignee: Marcus Eriksson Fix For: 2.2.0 rc1 About 1 out of 5 times, in a ccm-managed cluster, a repair doesn't write anything to the {{system_distributed.parent_repair_history}} table. [This dtest|https://github.com/riptano/cassandra-dtest/blob/master/repair_test.py#L296] reproduces the error on a 5-node cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[ https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569684#comment-14569684 ] Benjamin Lerer commented on CASSANDRA-9531: --- I tested the patch on my machine. It fixes the issue. +1 Thanks NullPointerException logged when running MessagePayloadTest --- Key: CASSANDRA-9531 URL: https://issues.apache.org/jira/browse/CASSANDRA-9531 Project: Cassandra Issue Type: Bug Reporter: Benjamin Lerer Assignee: T Jake Luciani Priority: Minor Fix For: 2.2.0 rc1 When I run the {{MessagePayloadTest}} I have the following error in my log output: {code} juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache processPendingNotifications AVERTISSEMENT: Exception thrown by removal listener java.lang.NullPointerException at org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260) at org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105) at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954) at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457) at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433) at com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104) at com.google.common.cache.LocalCache.remove(LocalCache.java:4170) at com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762) at org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182) at org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77) at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285) at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164) at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89) at org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45) at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} The problem is due to the fact that when {{getTotalBufferSize}} is called the segment has already been desallocated. The reason being that the segment has been put twice in the queue of the {{CacheBucket}}. I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2 calls to the {{close}} method of {{BigTableScanner}}. One here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207) {code} and the other here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[ https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569755#comment-14569755 ] Antoine Levy-Lambert commented on CASSANDRA-5543: - the interesting part in the attachment is that : {code} [java] error(208): /home/ubuntu/cassandra/src/java/org/apache/cassandra/cql/Cql.g:654:1: The following token definitions can never be matched because prior tokens match the same input: T__93,T__94,T__97,T__98,T__101,T__105,T__107,K_WITH,K_USING,K_USE,K_FIRST,K_COUNT,K_SET,K_APPLY,K_BATCH,K_TRUNCATE,K_IN,K_CREATE,K_KEYSPACE,K_COLUMNFAMILY,K_INDEX,K_ON,K_DROP,K_INTO,K_TIMESTAMP,K_TTL,K_ALTER,K_ADD,K_TYPE,RANGEOP,FLOAT,COMPIDENT,UUID,MULTILINE_COMMENT {code} I do not know antlr (I think this is some antlr generation). Someone should study this Cql.g to see whether it indeed does not comply with antlr's expectations and what to do about that. The stack trace above is just saying something like ant was executing an external java program with the java task, and the external java program failed You could change the build file to make the java/ invocation not fail on error but this might be hiding the problem. Ant issues when building gen-cql2-grammar - Key: CASSANDRA-5543 URL: https://issues.apache.org/jira/browse/CASSANDRA-5543 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.3 Reporter: Joaquin Casares Assignee: Dave Brosius Priority: Trivial Attachments: 5543-ant.log Below are the commands and outputs that were returned. The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant realclean` then it works fine after a second pass. {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml clean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/classes [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java [delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr realclean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build BUILD SUCCESSFUL Total time: 0 seconds {CODE} {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml maven-ant-tasks-localrepo: maven-ant-tasks-download: [echo] Downloading Maven ANT Tasks... [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar [get] To: /home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar maven-ant-tasks-init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib maven-declare-dependencies: maven-ant-tasks-retrieve-build: [artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from repository central at http://repo1.maven.org/maven2 [artifact:dependencies] [INFO] Unable to find resource 'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2 (http://download.java.net/maven/2) [artifact:dependencies] Building ant file: /home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml [copy] Copying 45 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/jars [copy] Copying 35 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/sources init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java check-avro-generate: avro-interface-generate-internode: [echo] Generating Avro internode code... avro-generate: build-subprojects: check-gen-cli-grammar: gen-cli-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g check-gen-cql2-grammar: gen-cql2-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g ... [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as IDENT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as K_KEY using multiple alternatives: 1, 2
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[ https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569774#comment-14569774 ] Antoine Levy-Lambert commented on CASSANDRA-5543: - I would focus on the error happening the first time that one builds. the second time that one builds antlr might be in no-op mode because it sees output files. in that case setting failonerror = false on the java task in the ant build file might be an acceptable workaround. but best to see what is the offending part in the grammar file. Ant issues when building gen-cql2-grammar - Key: CASSANDRA-5543 URL: https://issues.apache.org/jira/browse/CASSANDRA-5543 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.3 Reporter: Joaquin Casares Assignee: Dave Brosius Priority: Trivial Attachments: 5543-ant.log Below are the commands and outputs that were returned. The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant realclean` then it works fine after a second pass. {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml clean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/classes [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java [delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr realclean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build BUILD SUCCESSFUL Total time: 0 seconds {CODE} {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml maven-ant-tasks-localrepo: maven-ant-tasks-download: [echo] Downloading Maven ANT Tasks... [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar [get] To: /home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar maven-ant-tasks-init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib maven-declare-dependencies: maven-ant-tasks-retrieve-build: [artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from repository central at http://repo1.maven.org/maven2 [artifact:dependencies] [INFO] Unable to find resource 'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2 (http://download.java.net/maven/2) [artifact:dependencies] Building ant file: /home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml [copy] Copying 45 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/jars [copy] Copying 35 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/sources init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java check-avro-generate: avro-interface-generate-internode: [echo] Generating Avro internode code... avro-generate: build-subprojects: check-gen-cli-grammar: gen-cli-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g check-gen-cql2-grammar: gen-cql2-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g ... [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as IDENT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as K_KEY using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as QMARK using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as FLOAT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200):
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569721#comment-14569721 ] Brent Haines commented on CASSANDRA-9486: - FWIW -- I created a patch from [~slebresne]'s proposed fix, tested it locally and tried it on one of our impacted nodes in our data cluster. We are running 2.1.5, but the variance was easy to sift through. Compactions that used to take a day finished in 30 seconds. We haven't seen any spikes in heap space either, but that will take more time to evaluate. I do not know if there are any residual or unexpected behaviors resulting from the patch yet. We'll leave it in place for a few days to see. In the mean time, I thought it might help to confirm that this does indeed seem to fix our issue. Now, if we can stop creating so many RT's in the first place... :) LazilyCompactedRow accumulates all expired RangeTombstones -- Key: CASSANDRA-9486 URL: https://issues.apache.org/jira/browse/CASSANDRA-9486 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Assignee: Sylvain Lebresne Priority: Critical Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x Attachments: 0001-9486.patch LazilyCompactedRow initializes a ColumnIndex.Builder to use its RangeTombstone.Tracker, but it only calls update() with a RT argument, never an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So all the RT ever seen for the partition (that have expired) remain in memory until the compaction completes. To make matters worse, this then forces a linear scan of all of these RT for each live cell we add, so this extra load hangs around for a long time, and compactions stall. This issue is biting one of our users badly (at least, it seems likely to be this issue), and there may be others. This user is not even making use of RT extensively themselves, only collections (presumably with a complete overwrite of the contents of the collection, resulting in a RT being generated). Probably the best solution is to make the RT addition itself remove any already present that are no longer helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[ https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569779#comment-14569779 ] Mike Lococo commented on CASSANDRA-5543: Interesting speculation as to why subsequent runs may be working, that didn't occur to me. If it's not obvious, I have no antlr background either. I'll be watching this ticket and am happy to provide additional data or direct access to a box that reproduces this bug, but won't be digging into that potentially faulty grammar to find and fix the bug. Ant issues when building gen-cql2-grammar - Key: CASSANDRA-5543 URL: https://issues.apache.org/jira/browse/CASSANDRA-5543 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.3 Reporter: Joaquin Casares Assignee: Dave Brosius Priority: Trivial Attachments: 5543-ant.log Below are the commands and outputs that were returned. The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant realclean` then it works fine after a second pass. {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml clean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/classes [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java [delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr realclean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build BUILD SUCCESSFUL Total time: 0 seconds {CODE} {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml maven-ant-tasks-localrepo: maven-ant-tasks-download: [echo] Downloading Maven ANT Tasks... [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar [get] To: /home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar maven-ant-tasks-init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib maven-declare-dependencies: maven-ant-tasks-retrieve-build: [artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from repository central at http://repo1.maven.org/maven2 [artifact:dependencies] [INFO] Unable to find resource 'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2 (http://download.java.net/maven/2) [artifact:dependencies] Building ant file: /home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml [copy] Copying 45 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/jars [copy] Copying 35 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/sources init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java check-avro-generate: avro-interface-generate-internode: [echo] Generating Avro internode code... avro-generate: build-subprojects: check-gen-cli-grammar: gen-cli-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g check-gen-cql2-grammar: gen-cql2-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g ... [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as IDENT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as K_KEY using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as QMARK using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as FLOAT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200):
[jira] [Updated] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-9527: -- Attachment: 9527.txt The patch make sure that we log a different error message for this specific case. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x Attachments: 9527.txt In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8576) Primary Key Pushdown For Hadoop
[ https://issues.apache.org/jira/browse/CASSANDRA-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569713#comment-14569713 ] Philip Thompson commented on CASSANDRA-8576: This does not break any of the existing pig tests. I ran some additional tests, and found no major issues. As far as a mixed version cluster, I spun up a 3 node cluster of C*, with two nodes running this patch, the third without. I connected Pig to the cluster, using the unmodified node as the initial address. I then performed some map reduce jobs to select data from the cluster. The jobs succeeded, and I did see any errors in the log. +1 from me. Primary Key Pushdown For Hadoop --- Key: CASSANDRA-8576 URL: https://issues.apache.org/jira/browse/CASSANDRA-8576 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Russell Alexander Spitzer Assignee: Alex Liu Fix For: 2.1.x Attachments: 8576-2.1-branch.txt, 8576-trunk.txt, CASSANDRA-8576-v2-2.1-branch.txt, CASSANDRA-8576-v3-2.1-branch.txt I've heard reports from several users that they would like to have predicate pushdown functionality for hadoop (Hive in particular) based services. Example usecase Table with wide partitions, one per customer Application team has HQL they would like to run on a single customer Currently time to complete scales with number of customers since Input Format can't pushdown primary key predicate Current implementation requires a full table scan (since it can't recognize that a single partition was specified) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9534) Repair data not always saved to system_distributed
[ https://issues.apache.org/jira/browse/CASSANDRA-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Witschey updated CASSANDRA-9534: Summary: Repair data not always saved to system_distributed (was: Parent repair data not always saved to system_distributed) Repair data not always saved to system_distributed -- Key: CASSANDRA-9534 URL: https://issues.apache.org/jira/browse/CASSANDRA-9534 Project: Cassandra Issue Type: Bug Reporter: Jim Witschey Assignee: Marcus Eriksson Fix For: 2.2.0 rc1 About 1 out of 5 times, in a ccm-managed cluster, a repair doesn't write anything to the {{system_distributed.parent_repair_history}} table. [This dtest|https://github.com/riptano/cassandra-dtest/blob/master/repair_test.py#L296] reproduces the error on a 5-node cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5543) Ant issues when building gen-cql2-grammar
[ https://issues.apache.org/jira/browse/CASSANDRA-5543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569767#comment-14569767 ] Mike Lococo commented on CASSANDRA-5543: To be clear, it also works if you just run ant a second time with no additional changes. So... $ ant # error as noted above $ ant # works $ ant clean # works $ ant # same error $ ant # works The main thing this hurts is automated workflows on top of the ant build. I'm not sure how much of that is going on in the wild. I'm working around this for now by omitting the source build from the automated workflow I'm building. In my case it's not essential. I did want to drop the extra details I've found into this bug report, though, since it seems like it's been an irritation for some time. Ant issues when building gen-cql2-grammar - Key: CASSANDRA-5543 URL: https://issues.apache.org/jira/browse/CASSANDRA-5543 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.3 Reporter: Joaquin Casares Assignee: Dave Brosius Priority: Trivial Attachments: 5543-ant.log Below are the commands and outputs that were returned. The first `ant` command fails on gen-cql2-grammar, but if I don't run `ant realclean` then it works fine after a second pass. {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant realclean Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml clean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/test [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build/classes [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/src/gen-java [delete] Deleting: /home/ubuntu/.ccm/repository/1.2.3/build/internode.avpr realclean: [delete] Deleting directory /home/ubuntu/.ccm/repository/1.2.3/build BUILD SUCCESSFUL Total time: 0 seconds {CODE} {CODE} ubuntu@ip-10-196-153-29:~/.ccm/repository/1.2.3$ ant Buildfile: /home/ubuntu/.ccm/repository/1.2.3/build.xml maven-ant-tasks-localrepo: maven-ant-tasks-download: [echo] Downloading Maven ANT Tasks... [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build [get] Getting: http://repo2.maven.org/maven2/org/apache/maven/maven-ant-tasks/2.1.3/maven-ant-tasks-2.1.3.jar [get] To: /home/ubuntu/.ccm/repository/1.2.3/build/maven-ant-tasks-2.1.3.jar maven-ant-tasks-init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/lib maven-declare-dependencies: maven-ant-tasks-retrieve-build: [artifact:dependencies] Downloading: asm/asm/3.2/asm-3.2-sources.jar from repository central at http://repo1.maven.org/maven2 [artifact:dependencies] [INFO] Unable to find resource 'hsqldb:hsqldb:java-source:sources:1.8.0.10' in repository java.net2 (http://download.java.net/maven/2) [artifact:dependencies] Building ant file: /home/ubuntu/.ccm/repository/1.2.3/build/build-dependencies.xml [copy] Copying 45 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/jars [copy] Copying 35 files to /home/ubuntu/.ccm/repository/1.2.3/build/lib/sources init: [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/main [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/classes/thrift [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/lib [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/build/test/classes [mkdir] Created dir: /home/ubuntu/.ccm/repository/1.2.3/src/gen-java check-avro-generate: avro-interface-generate-internode: [echo] Generating Avro internode code... avro-generate: build-subprojects: check-gen-cli-grammar: gen-cli-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cli/Cli.g check-gen-cql2-grammar: gen-cql2-grammar: [echo] Building Grammar /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g ... [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as IDENT using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as K_KEY using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200): /home/ubuntu/.ccm/repository/1.2.3/src/java/org/apache/cassandra/cql/Cql.g:479:20: Decision can match input such as QMARK using multiple alternatives: 1, 2 [java] As a result, alternative(s) 2 were disabled for that input [java] warning(200):
[jira] [Commented] (CASSANDRA-9409) Ensure that UDF and UDAs are keyspace-isolated
[ https://issues.apache.org/jira/browse/CASSANDRA-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570009#comment-14570009 ] Zachary Kurey commented on CASSANDRA-9409: -- Is it expected that built in functions cannot be resolved by a UDA unless we specify the system keyspace? For example: {code} CREATE AGGREGATE if not exists ks.my_agg(int) SFUNC ks.adder STYPE int FINALFUNC intasblob ; {code} Gives me me the error: {quote}Final function ks.intasblob(int) does not exist or is not a scalar function{quote} Prefixing with 'system.' fixes the issue. But I was expecting it to resolve as if I were using a built in function elsewhere. Ensure that UDF and UDAs are keyspace-isolated -- Key: CASSANDRA-9409 URL: https://issues.apache.org/jira/browse/CASSANDRA-9409 Project: Cassandra Issue Type: Bug Reporter: Aleksey Yeschenko Assignee: Robert Stupp Labels: qa-resolved Fix For: 2.2.0 rc1 Attachments: 9409-with-system.txt, 9409.txt In table columns we don't allow to use UDTs from other keyspaces. We should also make sure that the following is *not* allowed: - UDFs taking UDTs from other keyspaces as arguments or return types - UDAs using UDFs from other keyspaces as its subfunctions The only exception should be made for {{system}} keyspace. UDAs and UDFs from any keyspace should be able to reference and reuse those. Having no dependencies between keyspaces makes this consistent with the way we treat UDTs, which is important, but also simplifies auth in multi-tenant environments, and is also crucial to upcoming 3.X strongly consistent schema work (the ability to treat keyspace as a unit of change and not having to worry about cross-keyspace dependencies). P.S. Should *probably* still allow using other keyspaces UDFs and UDAs in {{SELECT}} statements. Strict isolation in schema dependencies is what matters to me here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[ https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569894#comment-14569894 ] Ron Kuris commented on CASSANDRA-9536: -- One additional thought here is that it's probably fairly safe to record some extra 2 values since that errs on the side of not bringing the node down as fast. In fact, the first sample received is automatically recorded at value 2 so it's kind of a known quantity. The average quickly drops as values close to 1 are added to the list. The failure detector becomes more sensitive when the network is flakey -- Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Priority: Minor Fix For: 2.2.x I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The mean actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9536) The failure detector becomes more sensitive when the network is flakey
[ https://issues.apache.org/jira/browse/CASSANDRA-9536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9536: -- Fix Version/s: 2.2.x The failure detector becomes more sensitive when the network is flakey -- Key: CASSANDRA-9536 URL: https://issues.apache.org/jira/browse/CASSANDRA-9536 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Priority: Minor Fix For: 2.2.x I added considerable instrumentation into the failure detector, and then blocked port 7000 for a random 5-6 second interval, then resumed traffic for the same amount of time, with a script like: {code}while : do iptables -A INPUT -p tcp --destination-port 7000 -j DROP v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo offline for $s sleep $s iptables -F v=$[100 + (RANDOM % 100)]$[1000 + (RANDOM % 1000)] s=5.${v:1:2}${v:4:3} echo online for $s sleep $s done{code} When I do this, I watch the values being reported to the FailureDetector. The mean actually goes down, as low as 850ms. The reason is that the very slow packets are not recorded (they exceed MAX_INTERVAL_IN_NANO which is 2 seconds) and the retransmitted packets arrive very quickly in succession, lowering the overall average. Once the average is lowered, the node becomes much more sensitive to shorter outages. If you run this code for a while, the average drops down to 800ms or less, which means that the node will go down 20% quicker than expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[ https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihai Suteu updated CASSANDRA-8735: --- Attachment: CASSANDRA-8735.patch Created a patch for cassandra-2.2 The file location changed since the original issue was for cassandra 2.0.10, but the bug still persisted. Batch log replication is not randomized when there are only 2 racks --- Key: CASSANDRA-8735 URL: https://issues.apache.org/jira/browse/CASSANDRA-8735 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Priority: Minor Attachments: CASSANDRA-8735.patch Batch log replication is not randomized and the same 2 nodes can be picked up when there are only 2 racks in the cluster. https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[ https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihai Suteu reassigned CASSANDRA-8735: -- Assignee: Mihai Suteu Batch log replication is not randomized when there are only 2 racks --- Key: CASSANDRA-8735 URL: https://issues.apache.org/jira/browse/CASSANDRA-8735 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Assignee: Mihai Suteu Priority: Minor Attachments: CASSANDRA-8735.patch Batch log replication is not randomized and the same 2 nodes can be picked up when there are only 2 racks in the cluster. https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[ https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570189#comment-14570189 ] Mikhail Stepura commented on CASSANDRA-8735: [~yukim]. I don't remember. It looks like I made that test case based on the assumption that the behavior from Jonathan's patch CASSANDRA-6551 #2 *is* the expected one. Batch log replication is not randomized when there are only 2 racks --- Key: CASSANDRA-8735 URL: https://issues.apache.org/jira/browse/CASSANDRA-8735 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Assignee: Mihai Suteu Priority: Minor Fix For: 2.1.x, 2.2.x Attachments: CASSANDRA-8735.patch Batch log replication is not randomized and the same 2 nodes can be picked up when there are only 2 racks in the cluster. https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8735) Batch log replication is not randomized when there are only 2 racks
[ https://issues.apache.org/jira/browse/CASSANDRA-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570130#comment-14570130 ] Yuki Morishita commented on CASSANDRA-8735: --- So the patch will make {{BatchlogEndpointFilterTest#shouldSelectTwoFirstHostsFromSingleOtherRack}} fail occasionally since the test is expected to be always the first two. [~mishail] is this intended behavior? I saw the case where the same 2 nodes always getting a lot of batch log traffic. Batch log replication is not randomized when there are only 2 racks --- Key: CASSANDRA-8735 URL: https://issues.apache.org/jira/browse/CASSANDRA-8735 Project: Cassandra Issue Type: Bug Reporter: Yuki Morishita Assignee: Mihai Suteu Priority: Minor Fix For: 2.1.x, 2.2.x Attachments: CASSANDRA-8735.patch Batch log replication is not randomized and the same 2 nodes can be picked up when there are only 2 racks in the cluster. https://github.com/apache/cassandra/blob/cassandra-2.0.11/src/java/org/apache/cassandra/service/BatchlogEndpointSelector.java#L72-73 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9515) Bytes map encoding does not match protocol spec
[ https://issues.apache.org/jira/browse/CASSANDRA-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer updated CASSANDRA-9515: -- Attachment: 9515.txt The patch replace the {{byte[]}} type used for the values in the {{Map}} by the {{ByteBuffer}} type and use the {{readValue}} and {{writeValue}} methods from {{CBUtil}} to read and write the values. Bytes map encoding does not match protocol spec --- Key: CASSANDRA-9515 URL: https://issues.apache.org/jira/browse/CASSANDRA-9515 Project: Cassandra Issue Type: Bug Reporter: Olivier Michallat Assignee: Benjamin Lerer Fix For: 2.2.0 rc1 Attachments: 9515.txt In native_protocol_v4.spec, [bytes map] is defined as having [bytes] values, where [bytes] is an int n followed by n bytes. In practice, {{CBUtil#writeBytesMap}} encodes each value with {{CBUtil#writeBytes}}, which writes the length as a short (so a [short bytes] instead of a [bytes]). This is used for the custom payload introduced by CASSANDRA-8553. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8743) Repair on NFS in version 2.1.2
[ https://issues.apache.org/jira/browse/CASSANDRA-8743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568892#comment-14568892 ] Nicolas Guyomar commented on CASSANDRA-8743: HI, I just reproduced this 'exact' same issue on 2.1.5 while doing a repair (nodetool repair) after changing my cluster snitch My cluster is also on an NFS mounted drive I ran again the repair and got the same exception (keyspace name and table are voluntary changed in the stack) INFO [AntiEntropyStage:1] 2015-06-02 10:56:29,133 Validator.java:257 - [repair #412b6e80-0905-11e5-aa92-f327be1d597d] Sending completed merkle tree to /10.106.199.9 for mykeyspace/mytable2 ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,142 Validator.java:245 - Failed creating a merkle tree for [repair #412b6e80-0905-11e5-aa92-f327be1d597d on mykeyspace/mytable, (1997653793039594215,2135846523050696776]], /10.106.199.9 (see log for details) ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,158 CassandraDaemon.java:223 - Exception in thread Thread[ValidationExecutor:3,1,main] org.apache.cassandra.io.FSWriteError: java.nio.file.DirectoryNotEmptyException: /var/opt/data/flat/cassandra/mykeyspace/mytable-648d9b70abc011e49f8747338c540143/snapshots/412b6e80-0905-11e5-aa92-f327be1d597d at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:616) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:2341) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1013) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:95) ~[apache-cassandra-2.1.5.jar:2.1.5] at org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:617) ~[apache-cassandra-2.1.5.jar:2.1.5] at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.7.0_67] at java.lang.Thread.run(Unknown Source) [na:1.7.0_67] Caused by: java.nio.file.DirectoryNotEmptyException: /var/opt/data/flat/cassandra/mykeyspace/mytable-648d9b70abc011e49f8747338c540143/snapshots/412b6e80-0905-11e5-aa92-f327be1d597d at sun.nio.fs.UnixFileSystemProvider.implDelete(Unknown Source) ~[na:1.7.0_67] at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source) ~[na:1.7.0_67] at java.nio.file.Files.delete(Unknown Source) ~[na:1.7.0_67] at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) ~[apache-cassandra-2.1.5.jar:2.1.5] ... 10 common frames omitted ERROR [ValidationExecutor:3] 2015-06-02 10:56:29,166 StorageService.java:445 - Stopping gossiper WARN [ValidationExecutor:3] 2015-06-02 10:56:29,167 StorageService.java:351 - Stopping gossip by operator request INFO [ValidationExecutor:3] 2015-06-02 10:56:29,167 Gossiper.java:1410 - Announcing shutdown Repair on NFS in version 2.1.2 -- Key: CASSANDRA-8743 URL: https://issues.apache.org/jira/browse/CASSANDRA-8743 Project: Cassandra Issue Type: Bug Components: Core Reporter: Tamar Nirenberg Assignee: Joshua McKenzie Priority: Minor Running repair over NFS in Cassandra 2.1.2 encounters this error and crashes the ring: ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,811 Validator.java:232 - Failed creating a merkle tree for [repair #c84c7c70-a21b-11e4-aeca-19e6d7fa2595 on ATTRIBUTES/LINKS, (11621838520493020277529637175352775759,11853478749048239324667887059881170862]], /10.1.234.63 (see log for details) ERROR [ValidationExecutor:2] 2015-01-22 11:48:14,827 CassandraDaemon.java:153 - Exception in thread Thread[ValidationExecutor:2,1,main] org.apache.cassandra.io.FSWriteError: java.nio.file.DirectoryNotEmptyException: /exlibris/cassandra/local/data/data/ATTRIBUTES/LINKS/snapshots/c84c7c70-a21b-11e4-aeca-19e6d7fa2595 at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:381) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:547) ~[apache-cassandra-2.1.2.jar:2.1.2] at
[jira] [Updated] (CASSANDRA-9515) Bytes map encoding does not match protocol spec
[ https://issues.apache.org/jira/browse/CASSANDRA-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-9515: Reviewer: Sylvain Lebresne Bytes map encoding does not match protocol spec --- Key: CASSANDRA-9515 URL: https://issues.apache.org/jira/browse/CASSANDRA-9515 Project: Cassandra Issue Type: Bug Reporter: Olivier Michallat Assignee: Benjamin Lerer Fix For: 2.2.0 rc1 Attachments: 9515.txt In native_protocol_v4.spec, [bytes map] is defined as having [bytes] values, where [bytes] is an int n followed by n bytes. In practice, {{CBUtil#writeBytesMap}} encodes each value with {{CBUtil#writeBytes}}, which writes the length as a short (so a [short bytes] instead of a [bytes]). This is used for the custom payload introduced by CASSANDRA-8553. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7688) Add data sizing to a system table
[ https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568971#comment-14568971 ] Piotr Kołaczkowski edited comment on CASSANDRA-7688 at 6/2/15 11:29 AM: I double checked the cassandra.size_recorder_interval option does indeed work, but because many tests are creating fresh keyspaces.tables, the interval of 1 second is still too large and the test manages to run before the estimates are created. We'd need to put Thread.sleep before all the tests, which we're not going to do, because it would significantly increase runtime of the whole test suite. So far we just removed the warning about missing estimates. But it would be nice if C* filled those estimate entries on table creation (even with zeroes). There is a difference between we don't know the estimates and we know there are no data. was (Author: pkolaczk): I double checked the cassandra.size_recorder_interval option does indeed work, but because many tests are creating fresh keyspaces.tables, the interval of 1 second is still too large and the test manages to run before the estimates are created. We'd need to put Thread.sleep before all the tests, which we're not going to do, because it would So far we just removed the warning about missing estimates. But it would be nice if C* filled those estimate entries on table creation (even with zeroes). There is a difference between we don't know the estimates and we know there are no data. Add data sizing to a system table - Key: CASSANDRA-7688 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688 Project: Cassandra Issue Type: New Feature Reporter: Jeremiah Jordan Assignee: Aleksey Yeschenko Fix For: 2.1.5 Attachments: 7688.txt Currently you can't implement something similar to describe_splits_ex purely from the a native protocol driver. https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily getting ownership information to a client in the java-driver. But you still need the data sizing part to get splits of a given size. We should add the sizing information to a system table so that native clients can get to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569144#comment-14569144 ] Benjamin Lerer commented on CASSANDRA-9527: --- The issue is valid. Our test is: {code} if ((cfm.getCfDef().isCompact || !cfm.getCfDef().isComposite) cd.type != ColumnDefinition.Type.REGULAR) throw new InvalidRequestException(Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables); {code} The problem is that the type of the c column is {{ColumnDefinition.Type.COMPACT_VALUE}} and not {{ColumnDefinition.Type.REGULAR}}. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[ https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani reassigned CASSANDRA-9531: - Assignee: T Jake Luciani NullPointerException logged when running MessagePayloadTest --- Key: CASSANDRA-9531 URL: https://issues.apache.org/jira/browse/CASSANDRA-9531 Project: Cassandra Issue Type: Bug Reporter: Benjamin Lerer Assignee: T Jake Luciani Priority: Minor When I run the {{MessagePayloadTest}} I have the following error in my log output: {code} juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache processPendingNotifications AVERTISSEMENT: Exception thrown by removal listener java.lang.NullPointerException at org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260) at org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105) at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954) at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457) at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433) at com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104) at com.google.common.cache.LocalCache.remove(LocalCache.java:4170) at com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762) at org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182) at org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77) at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285) at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164) at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89) at org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45) at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} The problem is due to the fact that when {{getTotalBufferSize}} is called the segment has already been desallocated. The reason being that the segment has been put twice in the queue of the {{CacheBucket}}. I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2 calls to the {{close}} method of {{BigTableScanner}}. One here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207) {code} and the other here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9486) LazilyCompactedRow accumulates all expired RangeTombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569175#comment-14569175 ] Sylvain Lebresne commented on CASSANDRA-9486: - I had actually forgot one add in {{update}} which was breaking tests. I've pushed the fix and I'll wait once the tests have ran. bq. the semantics of an arbitrary method labelled insertBefore might well want to do so Right, felt more logical so makes hypothetical future misuse of the method less likely. LazilyCompactedRow accumulates all expired RangeTombstones -- Key: CASSANDRA-9486 URL: https://issues.apache.org/jira/browse/CASSANDRA-9486 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Assignee: Sylvain Lebresne Priority: Critical Fix For: 3.x, 2.1.x, 2.0.x, 2.2.x, 1.2.x Attachments: 0001-9486.patch LazilyCompactedRow initializes a ColumnIndex.Builder to use its RangeTombstone.Tracker, but it only calls update() with a RT argument, never an atom. The Tracker only ever _adds_ if it receives a RT, never removes. So all the RT ever seen for the partition (that have expired) remain in memory until the compaction completes. To make matters worse, this then forces a linear scan of all of these RT for each live cell we add, so this extra load hangs around for a long time, and compactions stall. This issue is biting one of our users badly (at least, it seems likely to be this issue), and there may be others. This user is not even making use of RT extensively themselves, only collections (presumably with a complete overwrite of the contents of the collection, resulting in a RT being generated). Probably the best solution is to make the RT addition itself remove any already present that are no longer helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9526) Provide a JMX hook to monitor phi values in the FailureDetector
[ https://issues.apache.org/jira/browse/CASSANDRA-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569127#comment-14569127 ] Brandon Williams commented on CASSANDRA-9526: - Thanks for the patch, Ron, this is good work. A couple of nits here: You disabled the logger.isTraceEnabled() in one call (which is fine) but then repeated the pattern you undid thereafter. I can be fine with either since they don't matter, but let's be consistent. Your IDE incorrectly reordered the imports. Provide a JMX hook to monitor phi values in the FailureDetector --- Key: CASSANDRA-9526 URL: https://issues.apache.org/jira/browse/CASSANDRA-9526 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Ron Kuris Fix For: 2.0.x Attachments: Monitor-Phi-JMX.patch.txt, PHI-Log-Debug-When-Close.patch.txt, PHI-Race-Condition.patch.txt phi_convict_threshold can be tuned, but there's currently no way to monitor the phi values to see if you're getting close. The attached patch adds the ability to get these values via JMX. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[ https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-9531: -- Fix Version/s: 2.2.0 rc1 NullPointerException logged when running MessagePayloadTest --- Key: CASSANDRA-9531 URL: https://issues.apache.org/jira/browse/CASSANDRA-9531 Project: Cassandra Issue Type: Bug Reporter: Benjamin Lerer Assignee: T Jake Luciani Priority: Minor Fix For: 2.2.0 rc1 When I run the {{MessagePayloadTest}} I have the following error in my log output: {code} juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache processPendingNotifications AVERTISSEMENT: Exception thrown by removal listener java.lang.NullPointerException at org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260) at org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105) at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954) at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457) at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433) at com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104) at com.google.common.cache.LocalCache.remove(LocalCache.java:4170) at com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762) at org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182) at org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77) at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285) at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164) at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89) at org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45) at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} The problem is due to the fact that when {{getTotalBufferSize}} is called the segment has already been desallocated. The reason being that the segment has been put twice in the queue of the {{CacheBucket}}. I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2 calls to the {{close}} method of {{BigTableScanner}}. One here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207) {code} and the other here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569201#comment-14569201 ] Stefania commented on CASSANDRA-8656: - I've attached a patch proposed by Marcus, that too has passed 2 rounds of 10 iterations on my box. Would you mind testing some more Michael? long-test LongLeveledCompactionStrategyTest flaps in 2.0 Key: CASSANDRA-8656 URL: https://issues.apache.org/jira/browse/CASSANDRA-8656 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Priority: Minor Labels: test-failure Fix For: 2.0.x Attachments: system.log LongLeveledCompactionStrategyTest periodically fails with: {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.412 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97) [junit] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409) [junit] Caused by: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) [junit] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) [junit] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [junit] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) [junit] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87) [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED {noformat} I would guess the failure is 10-20% of the time, looping over the test repeatedly. On the 2.1 branch, the failure is different, so perhaps this could also be updated. {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.04 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit]
[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table
[ https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568971#comment-14568971 ] Piotr Kołaczkowski commented on CASSANDRA-7688: --- I double checked the cassandra.size_recorder_interval option does indeed work, but because many tests are creating fresh keyspaces.tables, the interval of 1 second is still too large and the test manages to run before the estimates are created. We'd need to put Thread.sleep before all the tests, which we're not going to do, because it would So far we just removed the warning about missing estimates. But it would be nice if C* filled those estimate entries on table creation (even with zeroes). There is a difference between we don't know the estimates and we know there are no data. Add data sizing to a system table - Key: CASSANDRA-7688 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688 Project: Cassandra Issue Type: New Feature Reporter: Jeremiah Jordan Assignee: Aleksey Yeschenko Fix For: 2.1.5 Attachments: 7688.txt Currently you can't implement something similar to describe_splits_ex purely from the a native protocol driver. https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily getting ownership information to a client in the java-driver. But you still need the data sizing part to get splits of a given size. We should add the sizing information to a system table so that native clients can get to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569146#comment-14569146 ] Benjamin Lerer commented on CASSANDRA-9527: --- The issue is valid. Our test is: {code} if ((cfm.getCfDef().isCompact || !cfm.getCfDef().isComposite) cd.type != ColumnDefinition.Type.REGULAR) throw new InvalidRequestException(Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables); {code} The problem is that the type of the c column is {{ColumnDefinition.Type.COMPACT_VALUE}} and not {{ColumnDefinition.Type.REGULAR}}. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569151#comment-14569151 ] Michael Shuler commented on CASSANDRA-8656: --- I see there's a little irc discussion on this patch, but I've run 40 times over 2.0 HEAD + https://github.com/stef1927/cassandra/commit/6f7950e9db2ac7ad904e4962a9e5bd1cddfbb8e1.patch with no failures. long-test LongLeveledCompactionStrategyTest flaps in 2.0 Key: CASSANDRA-8656 URL: https://issues.apache.org/jira/browse/CASSANDRA-8656 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Priority: Minor Labels: test-failure Fix For: 2.0.x Attachments: system.log LongLeveledCompactionStrategyTest periodically fails with: {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.412 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97) [junit] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409) [junit] Caused by: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) [junit] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) [junit] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [junit] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) [junit] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87) [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED {noformat} I would guess the failure is 10-20% of the time, looping over the test repeatedly. On the 2.1 branch, the failure is different, so perhaps this could also be updated. {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.04 sec [junit] [junit] Testcase:
[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0f96e1d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0f96e1d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0f96e1d Branch: refs/heads/cassandra-2.1 Commit: c0f96e1d46d664a9d4ca7f982d21cdfcaed1a24a Parents: 14a3324 9b10928 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:51:20 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:51:20 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../cassandra/io/sstable/SSTableReader.java | 68 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- 4 files changed, 69 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/CHANGES.txt -- diff --cc CHANGES.txt index 71ce442,1aad965..4a7e174 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,37 -1,5 +1,38 @@@ -2.0.16: +2.1.6 + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index b784a7e,8b45005..498ad26 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -79,17 -95,18 +82,18 @@@ public class SSTableIdentityIterator im this.filename = filename; this.key = key; this.dataSize = dataSize; -this.expireBefore = (int)(System.currentTimeMillis() / 1000); this.flag = flag; this.validateColumns = checkData; -this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; + this.sstable = sstable; +Descriptor.Version dataVersion = sstable == null ?
[1/2] cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 14a332470 - c0f96e1d4 Always mark sstable suspected on corruption patch by slebresne; reviewed by benedict for CASSANDRA-9478 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c Branch: refs/heads/cassandra-2.1 Commit: 9b10928c159317160fb3049727679a48232b6041 Parents: 63819cb Author: Sylvain Lebresne sylv...@datastax.com Authored: Mon May 25 18:26:56 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:46:09 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 45 .../compaction/BlacklistingCompactionsTest.java | 16 --- 3 files changed, 48 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d23661d..1aad965 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 52da9bb..8b45005 100644 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat private final boolean validateColumns; private final String filename; +// Not every SSTableIdentifyIterator is attached to a sstable, so this can be null. +private final SSTableReader sstable; + /** * Used to iterate through the columns of a row. * @param sstable SSTable we are reading ffrom. @@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat this.flag = flag; this.validateColumns = checkData; this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +this.sstable = sstable; try { @@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { // catch here b/c atomIterator is an AbstractIterator; hasNext reads the value if (e.getCause() instanceof IOException) +{ +if (sstable != null) +sstable.markSuspect(); throw new CorruptSSTableException((IOException)e.getCause(), filename); +} else +{ throw e; +} } } @@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false); // since we already read column count, just pass that value and continue deserialization -IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); -while (iter.hasNext()) -cf.addAtom(iter.next()); +try +{ +IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); +while (iter.hasNext()) +cf.addAtom(iter.next()); -if (validateColumns) +if (validateColumns) +{ +try +{ +cf.metadata().validateColumns(cf); +} +catch (MarshalException e) +{ +throw new RuntimeException(Error validating row + key, e); +} +} +return cf; +} +catch (IOError e) { -try +// catch here b/c atomIterator is an AbstractIterator; hasNext reads the value +if (e.getCause() instanceof IOException) { -cf.metadata().validateColumns(cf); +if (sstable != null) +
[5/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Conflicts: src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a30d8bd2 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a30d8bd2 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a30d8bd2 Branch: refs/heads/trunk Commit: a30d8bd212a9a2fb4499130d2493eca34c3715a2 Parents: 94a68a1 c0f96e1 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:57:44 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:57:44 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 .../io/sstable/format/SSTableReader.java| 57 +++- .../compaction/BlacklistingCompactionsTest.java | 17 +++--- 4 files changed, 67 insertions(+), 18 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 6300749,498ad26..17f9a8d --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -78,10 -81,12 +81,11 @@@ import org.apache.cassandra.serializers this.in = in; this.filename = filename; this.key = key; -this.dataSize = dataSize; this.flag = flag; this.validateColumns = checkData; + this.sstable = sstable; -Descriptor.Version dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +Version dataVersion = sstable == null ? DatabaseDescriptor.getSSTableFormat().info.getLatestVersion() : sstable.descriptor.version; int expireBefore = (int) (System.currentTimeMillis() / 1000); columnFamily = ArrayBackedSortedColumns.factory.create(metadata);
[2/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0f96e1d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0f96e1d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0f96e1d Branch: refs/heads/trunk Commit: c0f96e1d46d664a9d4ca7f982d21cdfcaed1a24a Parents: 14a3324 9b10928 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:51:20 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:51:20 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../cassandra/io/sstable/SSTableReader.java | 68 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- 4 files changed, 69 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/CHANGES.txt -- diff --cc CHANGES.txt index 71ce442,1aad965..4a7e174 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,37 -1,5 +1,38 @@@ -2.0.16: +2.1.6 + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index b784a7e,8b45005..498ad26 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -79,17 -95,18 +82,18 @@@ public class SSTableIdentityIterator im this.filename = filename; this.key = key; this.dataSize = dataSize; -this.expireBefore = (int)(System.currentTimeMillis() / 1000); this.flag = flag; this.validateColumns = checkData; -this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; + this.sstable = sstable; +Descriptor.Version dataVersion = sstable == null ? Descriptor.Version.CURRENT :
[1/6] cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra Updated Branches: refs/heads/trunk 525579d0d - f90207c06 Always mark sstable suspected on corruption patch by slebresne; reviewed by benedict for CASSANDRA-9478 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c Branch: refs/heads/trunk Commit: 9b10928c159317160fb3049727679a48232b6041 Parents: 63819cb Author: Sylvain Lebresne sylv...@datastax.com Authored: Mon May 25 18:26:56 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:46:09 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 45 .../compaction/BlacklistingCompactionsTest.java | 16 --- 3 files changed, 48 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d23661d..1aad965 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 52da9bb..8b45005 100644 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat private final boolean validateColumns; private final String filename; +// Not every SSTableIdentifyIterator is attached to a sstable, so this can be null. +private final SSTableReader sstable; + /** * Used to iterate through the columns of a row. * @param sstable SSTable we are reading ffrom. @@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat this.flag = flag; this.validateColumns = checkData; this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +this.sstable = sstable; try { @@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { // catch here b/c atomIterator is an AbstractIterator; hasNext reads the value if (e.getCause() instanceof IOException) +{ +if (sstable != null) +sstable.markSuspect(); throw new CorruptSSTableException((IOException)e.getCause(), filename); +} else +{ throw e; +} } } @@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false); // since we already read column count, just pass that value and continue deserialization -IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); -while (iter.hasNext()) -cf.addAtom(iter.next()); +try +{ +IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); +while (iter.hasNext()) +cf.addAtom(iter.next()); -if (validateColumns) +if (validateColumns) +{ +try +{ +cf.metadata().validateColumns(cf); +} +catch (MarshalException e) +{ +throw new RuntimeException(Error validating row + key, e); +} +} +return cf; +} +catch (IOError e) { -try +// catch here b/c atomIterator is an AbstractIterator; hasNext reads the value +if (e.getCause() instanceof IOException) { -cf.metadata().validateColumns(cf); +if (sstable != null) +sstable.markSuspect();
[3/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java -- diff --cc test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java index 5420b1b,572ad36..2b6a62a --- a/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java +++ b/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java @@@ -22,11 -22,8 +22,9 @@@ package org.apache.cassandra.db.compact import java.io.RandomAccessFile; - import java.util.Collection; - import java.util.HashSet; - import java.util.Set; + import java.util.*; +import org.apache.cassandra.io.sstable.format.SSTableReader; import org.junit.After; import org.junit.AfterClass; import org.junit.BeforeClass; @@@ -43,12 -38,12 +41,13 @@@ import org.apache.cassandra.utils.ByteB import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; + import static org.junit.Assert.assertTrue; import static org.apache.cassandra.Util.cellname; -public class BlacklistingCompactionsTest extends SchemaLoader +public class BlacklistingCompactionsTest { -public static final String KEYSPACE = Keyspace1; +private static final String KEYSPACE1 = BlacklistingCompactionsTest; +private static final String CF_STANDARD1 = Standard1; @After public void leakDetect() throws InterruptedException
[6/6] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f90207c0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f90207c0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f90207c0 Branch: refs/heads/trunk Commit: f90207c0661812ad1dc6bc6d7672898406cf9ec0 Parents: 525579d a30d8bd Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:58:18 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:58:18 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 .../io/sstable/format/SSTableReader.java| 57 +++- .../compaction/BlacklistingCompactionsTest.java | 17 +++--- 4 files changed, 67 insertions(+), 18 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f90207c0/CHANGES.txt --
[4/6] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index ba634b6,000..f4a59a2 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@@ -1,2142 -1,0 +1,2175 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.io.sstable.format; + +import java.io.*; +import java.nio.ByteBuffer; +import java.util.*; +import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; + +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Predicate; +import com.google.common.collect.Iterators; +import com.google.common.collect.Ordering; +import com.google.common.primitives.Longs; +import com.google.common.util.concurrent.RateLimiter; + +import com.clearspring.analytics.stream.cardinality.CardinalityMergeException; +import com.clearspring.analytics.stream.cardinality.HyperLogLogPlus; +import com.clearspring.analytics.stream.cardinality.ICardinality; +import org.apache.cassandra.cache.CachingOptions; +import org.apache.cassandra.cache.InstrumentingCache; +import org.apache.cassandra.cache.KeyCacheKey; +import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor; +import org.apache.cassandra.concurrent.ScheduledExecutors; +import org.apache.cassandra.config.*; +import org.apache.cassandra.db.*; +import org.apache.cassandra.db.columniterator.OnDiskAtomIterator; +import org.apache.cassandra.db.commitlog.ReplayPosition; +import org.apache.cassandra.db.composites.CellName; +import org.apache.cassandra.db.filter.ColumnSlice; +import org.apache.cassandra.db.index.SecondaryIndex; +import org.apache.cassandra.db.lifecycle.Tracker; +import org.apache.cassandra.dht.*; +import org.apache.cassandra.io.compress.CompressionMetadata; +import org.apache.cassandra.io.sstable.*; +import org.apache.cassandra.io.sstable.metadata.*; +import org.apache.cassandra.io.util.*; +import org.apache.cassandra.metrics.RestorableMeter; +import org.apache.cassandra.metrics.StorageMetrics; +import org.apache.cassandra.service.ActiveRepairService; +import org.apache.cassandra.service.CacheService; +import org.apache.cassandra.service.StorageService; +import org.apache.cassandra.utils.*; +import org.apache.cassandra.utils.concurrent.OpOrder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.apache.cassandra.utils.concurrent.Ref; +import org.apache.cassandra.utils.concurrent.SelfRefCounted; + +import static org.apache.cassandra.db.Directories.SECONDARY_INDEX_NAME_SEPARATOR; + +/** + * An SSTableReader can be constructed in a number of places, but typically is either + * read from disk at startup, or constructed from a flushed memtable, or after compaction + * to replace some existing sstables. However once created, an sstablereader may also be modified. + * + * A reader's OpenReason describes its current stage in its lifecycle, as follows: + * + * + * pre {@code + * NORMAL + * From: None= Reader has been read from disk, either at startup or from a flushed memtable + * EARLY = Reader is the final result of a compaction + * MOVED_START = Reader WAS being compacted, but this failed and it has been restored to NORMAL status + * + * EARLY + * From: None= Reader is a compaction replacement that is either incomplete and has been opened + *to represent its partial result status, or has been finished but the compaction + *it is a part of has not yet completed fully + * EARLY = Same as from None, only it is not the first time it has been + * + * MOVED_START + * From: NORMAL = Reader is being compacted. This compaction has not finished, but the compaction result + *
[jira] [Updated] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-9527: --- Assignee: Benjamin Lerer Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/2] cassandra git commit: Fix custom payload encoding decoding to match protocol spec
Repository: cassandra Updated Branches: refs/heads/trunk f90207c06 - ae2daea52 Fix custom payload encoding decoding to match protocol spec patch by blerer; reviewed by slebresne for CASSANDRA-9515 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f93bd1f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f93bd1f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f93bd1f Branch: refs/heads/trunk Commit: 6f93bd1f65888104e33da2f9f01056b6115952e5 Parents: a30d8bd Author: Benjamin Lerer benjamin.le...@datastax.com Authored: Tue Jun 2 15:34:31 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 15:34:31 2015 +0200 -- CHANGES.txt | 1 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../cassandra/transport/MessagePayloadTest.java | 73 7 files changed, 110 insertions(+), 55 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 7f0ef51..db94c76 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2 + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411) http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java -- diff --git a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java index 3930e9c..02a6df9 100644 --- a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java +++ b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java @@ -17,6 +17,7 @@ */ package org.apache.cassandra.cql3; +import java.nio.ByteBuffer; import java.util.Map; import org.apache.cassandra.cql3.statements.BatchStatement; @@ -34,14 +35,17 @@ public class CustomPayloadMirroringQueryHandler implements QueryHandler { static QueryProcessor queryProcessor = QueryProcessor.instance; -public ResultMessage process(String query, QueryState state, QueryOptions options, MapString, byte[] customPayload) +public ResultMessage process(String query, + QueryState state, + QueryOptions options, + MapString, ByteBuffer customPayload) { ResultMessage result = queryProcessor.process(query, state, options, customPayload); result.setCustomPayload(customPayload); return result; } -public ResultMessage.Prepared prepare(String query, QueryState state, MapString, byte[] customPayload) +public ResultMessage.Prepared prepare(String query, QueryState state, MapString, ByteBuffer customPayload) { ResultMessage.Prepared prepared = queryProcessor.prepare(query, state, customPayload); prepared.setCustomPayload(customPayload); @@ -58,14 +62,20 @@ public class CustomPayloadMirroringQueryHandler implements QueryHandler return queryProcessor.getPreparedForThrift(id); } -public ResultMessage processPrepared(CQLStatement statement, QueryState state, QueryOptions options, MapString, byte[] customPayload) +public ResultMessage processPrepared(CQLStatement statement, + QueryState state, + QueryOptions options, + MapString, ByteBuffer customPayload) { ResultMessage result = queryProcessor.processPrepared(statement, state, options, customPayload); result.setCustomPayload(customPayload); return result; } -public ResultMessage processBatch(BatchStatement statement, QueryState state, BatchQueryOptions options, MapString, byte[] customPayload) +public ResultMessage processBatch(BatchStatement statement, + QueryState state, + BatchQueryOptions options, + MapString, ByteBuffer customPayload) { ResultMessage result = queryProcessor.processBatch(statement, state, options, customPayload);
[5/5] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e27e9741 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e27e9741 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e27e9741 Branch: refs/heads/trunk Commit: e27e974139d371c00c49ad70504f0283caaba356 Parents: ae2daea 4cfb17f Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:43:33 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:43:33 2015 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/gms/Gossiper.java | 16 -- .../cassandra/service/StorageService.java | 54 +--- 3 files changed, 26 insertions(+), 45 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e27e9741/CHANGES.txt -- diff --cc CHANGES.txt index 474476f,42846d7..4597175 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,10 -1,5 +1,11 @@@ +3.0: + * Decommissioned nodes will not rejoin the cluster (CASSANDRA-8801) + * Change gossip stabilization to use endpoit size (CASSANDRA-9401) + * Change default garbage collector to G1 (CASSANDRA-7486) + + 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for http://git-wip-us.apache.org/repos/asf/cassandra/blob/e27e9741/src/java/org/apache/cassandra/service/StorageService.java --
[2/5] cassandra git commit: Clean up gossiper logic for old versions
Clean up gossiper logic for old versions Patch by brandonwilliams, reviewed by aleksey for CASSANDRA-9370 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/23d7a558 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/23d7a558 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/23d7a558 Branch: refs/heads/trunk Commit: 23d7a558ee824a22d382e1b645814638a2cd2985 Parents: 94a68a1 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:42:28 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:42:28 2015 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/gms/Gossiper.java | 16 -- .../cassandra/service/StorageService.java | 54 +--- 3 files changed, 26 insertions(+), 45 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 165f76b..41f5cfb 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411) http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/gms/Gossiper.java -- diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java b/src/java/org/apache/cassandra/gms/Gossiper.java index 0c449ec..e131da6 100644 --- a/src/java/org/apache/cassandra/gms/Gossiper.java +++ b/src/java/org/apache/cassandra/gms/Gossiper.java @@ -798,24 +798,8 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean return endpointStateMap.entrySet(); } -public boolean usesHostId(InetAddress endpoint) -{ -if (MessagingService.instance().knowsVersion(endpoint)) -return true; -else if (getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.NET_VERSION) != null) -return true; -return false; -} - -public boolean usesVnodes(InetAddress endpoint) -{ -return usesHostId(endpoint) getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.TOKENS) != null; -} - public UUID getHostId(InetAddress endpoint) { -if (!usesHostId(endpoint)) -throw new RuntimeException(Host + endpoint + does not use new-style tokens!); return UUID.fromString(getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.HOST_ID).value); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index bfbf1a8..8afb93f 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -1825,8 +1825,7 @@ public class StorageService extends NotificationBroadcasterSupport implements IE tokenMetadata.addBootstrapTokens(tokens, endpoint); PendingRangeCalculatorService.instance.update(); -if (Gossiper.instance.usesHostId(endpoint)) -tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint), endpoint); +tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint), endpoint); } /** @@ -1855,39 +1854,36 @@ public class StorageService extends NotificationBroadcasterSupport implements IE updatePeerInfo(endpoint); // Order Matters, TM.updateHostID() should be called before TM.updateNormalToken(), (see CASSANDRA-4300). -if (Gossiper.instance.usesHostId(endpoint)) +UUID hostId = Gossiper.instance.getHostId(endpoint); +InetAddress existing = tokenMetadata.getEndpointForHostId(hostId); +if (replacing Gossiper.instance.getEndpointStateForEndpoint(DatabaseDescriptor.getReplaceAddress()) != null (hostId.equals(Gossiper.instance.getHostId(DatabaseDescriptor.getReplaceAddress() +logger.warn(Not updating token metadata for {} because I am replacing it, endpoint); +else { -UUID hostId = Gossiper.instance.getHostId(endpoint); -InetAddress existing = tokenMetadata.getEndpointForHostId(hostId); -if (replacing Gossiper.instance.getEndpointStateForEndpoint(DatabaseDescriptor.getReplaceAddress()) != null
[jira] [Updated] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-9527: --- Reproduced In: 2.0.13 Fix Version/s: 2.0.x Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ae2daea5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ae2daea5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ae2daea5 Branch: refs/heads/trunk Commit: ae2daea52456bd460663d7aecff9180810063834 Parents: f90207c 6f93bd1 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 15:35:37 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 15:35:37 2015 +0200 -- CHANGES.txt | 1 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../cassandra/transport/MessagePayloadTest.java | 73 7 files changed, 110 insertions(+), 55 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae2daea5/CHANGES.txt -- diff --cc CHANGES.txt index 59bdadc,db94c76..474476f --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,10 -1,5 +1,11 @@@ +3.0: + * Decommissioned nodes will not rejoin the cluster (CASSANDRA-8801) + * Change gossip stabilization to use endpoit size (CASSANDRA-9401) + * Change default garbage collector to G1 (CASSANDRA-7486) + + 2.2 + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411)
cassandra git commit: Fix custom payload encoding decoding to match protocol spec
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 a30d8bd21 - 6f93bd1f6 Fix custom payload encoding decoding to match protocol spec patch by blerer; reviewed by slebresne for CASSANDRA-9515 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6f93bd1f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6f93bd1f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6f93bd1f Branch: refs/heads/cassandra-2.2 Commit: 6f93bd1f65888104e33da2f9f01056b6115952e5 Parents: a30d8bd Author: Benjamin Lerer benjamin.le...@datastax.com Authored: Tue Jun 2 15:34:31 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 15:34:31 2015 +0200 -- CHANGES.txt | 1 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../cassandra/transport/MessagePayloadTest.java | 73 7 files changed, 110 insertions(+), 55 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 7f0ef51..db94c76 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2 + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411) http://git-wip-us.apache.org/repos/asf/cassandra/blob/6f93bd1f/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java -- diff --git a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java index 3930e9c..02a6df9 100644 --- a/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java +++ b/src/java/org/apache/cassandra/cql3/CustomPayloadMirroringQueryHandler.java @@ -17,6 +17,7 @@ */ package org.apache.cassandra.cql3; +import java.nio.ByteBuffer; import java.util.Map; import org.apache.cassandra.cql3.statements.BatchStatement; @@ -34,14 +35,17 @@ public class CustomPayloadMirroringQueryHandler implements QueryHandler { static QueryProcessor queryProcessor = QueryProcessor.instance; -public ResultMessage process(String query, QueryState state, QueryOptions options, MapString, byte[] customPayload) +public ResultMessage process(String query, + QueryState state, + QueryOptions options, + MapString, ByteBuffer customPayload) { ResultMessage result = queryProcessor.process(query, state, options, customPayload); result.setCustomPayload(customPayload); return result; } -public ResultMessage.Prepared prepare(String query, QueryState state, MapString, byte[] customPayload) +public ResultMessage.Prepared prepare(String query, QueryState state, MapString, ByteBuffer customPayload) { ResultMessage.Prepared prepared = queryProcessor.prepare(query, state, customPayload); prepared.setCustomPayload(customPayload); @@ -58,14 +62,20 @@ public class CustomPayloadMirroringQueryHandler implements QueryHandler return queryProcessor.getPreparedForThrift(id); } -public ResultMessage processPrepared(CQLStatement statement, QueryState state, QueryOptions options, MapString, byte[] customPayload) +public ResultMessage processPrepared(CQLStatement statement, + QueryState state, + QueryOptions options, + MapString, ByteBuffer customPayload) { ResultMessage result = queryProcessor.processPrepared(statement, state, options, customPayload); result.setCustomPayload(customPayload); return result; } -public ResultMessage processBatch(BatchStatement statement, QueryState state, BatchQueryOptions options, MapString, byte[] customPayload) +public ResultMessage processBatch(BatchStatement statement, + QueryState state, + BatchQueryOptions options, + MapString, ByteBuffer customPayload) { ResultMessage result = queryProcessor.processBatch(statement, state, options,
[2/5] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: CHANGES.txt src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c0f96e1d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c0f96e1d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c0f96e1d Branch: refs/heads/cassandra-2.2 Commit: c0f96e1d46d664a9d4ca7f982d21cdfcaed1a24a Parents: 14a3324 9b10928 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:51:20 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:51:20 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../cassandra/io/sstable/SSTableReader.java | 68 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- 4 files changed, 69 insertions(+), 27 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/CHANGES.txt -- diff --cc CHANGES.txt index 71ce442,1aad965..4a7e174 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,37 -1,5 +1,38 @@@ -2.0.16: +2.1.6 + * Consistent error message when a table mixes counter and non-counter + columns (CASSANDRA-9492) + * Avoid getting unreadable keys during anticompaction (CASSANDRA-9508) + * (cqlsh) Better float precision by default (CASSANDRA-9224) + * Improve estimated row count (CASSANDRA-9107) + * Optimize range tombstone memory footprint (CASSANDRA-8603) + * Use configured gcgs in anticompaction (CASSANDRA-9397) + * Warn on misuse of unlogged batches (CASSANDRA-9282) + * Failure detector detects and ignores local pauses (CASSANDRA-9183) + * Add utility class to support for rate limiting a given log statement (CASSANDRA-9029) + * Add missing consistency levels to cassandra-stess (CASSANDRA-9361) + * Fix commitlog getCompletedTasks to not increment (CASSANDRA-9339) + * Fix for harmless exceptions logged as ERROR (CASSANDRA-8564) + * Delete processed sstables in sstablesplit/sstableupgrade (CASSANDRA-8606) + * Improve sstable exclusion from partition tombstones (CASSANDRA-9298) + * Validate the indexed column rather than the cell's contents for 2i (CASSANDRA-9057) + * Add support for top-k custom 2i queries (CASSANDRA-8717) + * Fix error when dropping table during compaction (CASSANDRA-9251) + * cassandra-stress supports validation operations over user profiles (CASSANDRA-8773) + * Add support for rate limiting log messages (CASSANDRA-9029) + * Log the partition key with tombstone warnings (CASSANDRA-8561) + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) + * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261) + * Fix anticompaction blocking ANTI_ENTROPY stage (CASSANDRA-9151) + * Repair waits for anticompaction to finish (CASSANDRA-9097) + * Fix streaming not holding ref when stream error (CASSANDRA-9295) + * Fix canonical view returning early opened SSTables (CASSANDRA-9396) +Merged from 2.0: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c0f96e1d/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index b784a7e,8b45005..498ad26 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -79,17 -95,18 +82,18 @@@ public class SSTableIdentityIterator im this.filename = filename; this.key = key; this.dataSize = dataSize; -this.expireBefore = (int)(System.currentTimeMillis() / 1000); this.flag = flag; this.validateColumns = checkData; -this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; + this.sstable = sstable; +Descriptor.Version dataVersion = sstable == null ?
[5/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
Merge branch 'cassandra-2.1' into cassandra-2.2 Conflicts: src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a30d8bd2 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a30d8bd2 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a30d8bd2 Branch: refs/heads/cassandra-2.2 Commit: a30d8bd212a9a2fb4499130d2493eca34c3715a2 Parents: 94a68a1 c0f96e1 Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jun 2 14:57:44 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:57:44 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 10 .../io/sstable/format/SSTableReader.java| 57 +++- .../compaction/BlacklistingCompactionsTest.java | 17 +++--- 4 files changed, 67 insertions(+), 18 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --cc src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 6300749,498ad26..17f9a8d --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@@ -78,10 -81,12 +81,11 @@@ import org.apache.cassandra.serializers this.in = in; this.filename = filename; this.key = key; -this.dataSize = dataSize; this.flag = flag; this.validateColumns = checkData; + this.sstable = sstable; -Descriptor.Version dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +Version dataVersion = sstable == null ? DatabaseDescriptor.getSSTableFormat().info.getLatestVersion() : sstable.descriptor.version; int expireBefore = (int) (System.currentTimeMillis() / 1000); columnFamily = ArrayBackedSortedColumns.factory.create(metadata);
[4/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java -- diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java index ba634b6,000..f4a59a2 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java +++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java @@@ -1,2142 -1,0 +1,2175 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.io.sstable.format; + +import java.io.*; +import java.nio.ByteBuffer; +import java.util.*; +import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; + +import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Predicate; +import com.google.common.collect.Iterators; +import com.google.common.collect.Ordering; +import com.google.common.primitives.Longs; +import com.google.common.util.concurrent.RateLimiter; + +import com.clearspring.analytics.stream.cardinality.CardinalityMergeException; +import com.clearspring.analytics.stream.cardinality.HyperLogLogPlus; +import com.clearspring.analytics.stream.cardinality.ICardinality; +import org.apache.cassandra.cache.CachingOptions; +import org.apache.cassandra.cache.InstrumentingCache; +import org.apache.cassandra.cache.KeyCacheKey; +import org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor; +import org.apache.cassandra.concurrent.ScheduledExecutors; +import org.apache.cassandra.config.*; +import org.apache.cassandra.db.*; +import org.apache.cassandra.db.columniterator.OnDiskAtomIterator; +import org.apache.cassandra.db.commitlog.ReplayPosition; +import org.apache.cassandra.db.composites.CellName; +import org.apache.cassandra.db.filter.ColumnSlice; +import org.apache.cassandra.db.index.SecondaryIndex; +import org.apache.cassandra.db.lifecycle.Tracker; +import org.apache.cassandra.dht.*; +import org.apache.cassandra.io.compress.CompressionMetadata; +import org.apache.cassandra.io.sstable.*; +import org.apache.cassandra.io.sstable.metadata.*; +import org.apache.cassandra.io.util.*; +import org.apache.cassandra.metrics.RestorableMeter; +import org.apache.cassandra.metrics.StorageMetrics; +import org.apache.cassandra.service.ActiveRepairService; +import org.apache.cassandra.service.CacheService; +import org.apache.cassandra.service.StorageService; +import org.apache.cassandra.utils.*; +import org.apache.cassandra.utils.concurrent.OpOrder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.apache.cassandra.utils.concurrent.Ref; +import org.apache.cassandra.utils.concurrent.SelfRefCounted; + +import static org.apache.cassandra.db.Directories.SECONDARY_INDEX_NAME_SEPARATOR; + +/** + * An SSTableReader can be constructed in a number of places, but typically is either + * read from disk at startup, or constructed from a flushed memtable, or after compaction + * to replace some existing sstables. However once created, an sstablereader may also be modified. + * + * A reader's OpenReason describes its current stage in its lifecycle, as follows: + * + * + * pre {@code + * NORMAL + * From: None= Reader has been read from disk, either at startup or from a flushed memtable + * EARLY = Reader is the final result of a compaction + * MOVED_START = Reader WAS being compacted, but this failed and it has been restored to NORMAL status + * + * EARLY + * From: None= Reader is a compaction replacement that is either incomplete and has been opened + *to represent its partial result status, or has been finished but the compaction + *it is a part of has not yet completed fully + * EARLY = Same as from None, only it is not the first time it has been + * + * MOVED_START + * From: NORMAL = Reader is being compacted. This compaction has not finished, but the compaction result + *
[3/5] cassandra git commit: Merge branch 'cassandra-2.1' into cassandra-2.2
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a30d8bd2/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java -- diff --cc test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java index 5420b1b,572ad36..2b6a62a --- a/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java +++ b/test/unit/org/apache/cassandra/db/compaction/BlacklistingCompactionsTest.java @@@ -22,11 -22,8 +22,9 @@@ package org.apache.cassandra.db.compact import java.io.RandomAccessFile; - import java.util.Collection; - import java.util.HashSet; - import java.util.Set; + import java.util.*; +import org.apache.cassandra.io.sstable.format.SSTableReader; import org.junit.After; import org.junit.AfterClass; import org.junit.BeforeClass; @@@ -43,12 -38,12 +41,13 @@@ import org.apache.cassandra.utils.ByteB import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; + import static org.junit.Assert.assertTrue; import static org.apache.cassandra.Util.cellname; -public class BlacklistingCompactionsTest extends SchemaLoader +public class BlacklistingCompactionsTest { -public static final String KEYSPACE = Keyspace1; +private static final String KEYSPACE1 = BlacklistingCompactionsTest; +private static final String CF_STANDARD1 = Standard1; @After public void leakDetect() throws InterruptedException
[1/5] cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 94a68a17c - a30d8bd21 Always mark sstable suspected on corruption patch by slebresne; reviewed by benedict for CASSANDRA-9478 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c Branch: refs/heads/cassandra-2.2 Commit: 9b10928c159317160fb3049727679a48232b6041 Parents: 63819cb Author: Sylvain Lebresne sylv...@datastax.com Authored: Mon May 25 18:26:56 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:46:09 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 45 .../compaction/BlacklistingCompactionsTest.java | 16 --- 3 files changed, 48 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d23661d..1aad965 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 52da9bb..8b45005 100644 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat private final boolean validateColumns; private final String filename; +// Not every SSTableIdentifyIterator is attached to a sstable, so this can be null. +private final SSTableReader sstable; + /** * Used to iterate through the columns of a row. * @param sstable SSTable we are reading ffrom. @@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat this.flag = flag; this.validateColumns = checkData; this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +this.sstable = sstable; try { @@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { // catch here b/c atomIterator is an AbstractIterator; hasNext reads the value if (e.getCause() instanceof IOException) +{ +if (sstable != null) +sstable.markSuspect(); throw new CorruptSSTableException((IOException)e.getCause(), filename); +} else +{ throw e; +} } } @@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false); // since we already read column count, just pass that value and continue deserialization -IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); -while (iter.hasNext()) -cf.addAtom(iter.next()); +try +{ +IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); +while (iter.hasNext()) +cf.addAtom(iter.next()); -if (validateColumns) +if (validateColumns) +{ +try +{ +cf.metadata().validateColumns(cf); +} +catch (MarshalException e) +{ +throw new RuntimeException(Error validating row + key, e); +} +} +return cf; +} +catch (IOError e) { -try +// catch here b/c atomIterator is an AbstractIterator; hasNext reads the value +if (e.getCause() instanceof IOException) { -cf.metadata().validateColumns(cf); +if (sstable != null) +
cassandra git commit: Always mark sstable suspected on corruption
Repository: cassandra Updated Branches: refs/heads/cassandra-2.0 63819cbee - 9b10928c1 Always mark sstable suspected on corruption patch by slebresne; reviewed by benedict for CASSANDRA-9478 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b10928c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b10928c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b10928c Branch: refs/heads/cassandra-2.0 Commit: 9b10928c159317160fb3049727679a48232b6041 Parents: 63819cb Author: Sylvain Lebresne sylv...@datastax.com Authored: Mon May 25 18:26:56 2015 +0200 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jun 2 14:46:09 2015 +0200 -- CHANGES.txt | 1 + .../io/sstable/SSTableIdentityIterator.java | 45 .../compaction/BlacklistingCompactionsTest.java | 16 --- 3 files changed, 48 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d23661d..1aad965 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.0.16: + * Always mark sstable suspect when corrupted (CASSANDRA-9478) * Add database users and permissions to CQL3 documentation (CASSANDRA-7558) * Allow JVM_OPTS to be passed to standalone tools (CASSANDRA-5969) * Fix bad condition in RangeTombstoneList (CASSANDRA-9485) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b10928c/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java index 52da9bb..8b45005 100644 --- a/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java +++ b/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java @@ -50,6 +50,9 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat private final boolean validateColumns; private final String filename; +// Not every SSTableIdentifyIterator is attached to a sstable, so this can be null. +private final SSTableReader sstable; + /** * Used to iterate through the columns of a row. * @param sstable SSTable we are reading ffrom. @@ -96,6 +99,7 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat this.flag = flag; this.validateColumns = checkData; this.dataVersion = sstable == null ? Descriptor.Version.CURRENT : sstable.descriptor.version; +this.sstable = sstable; try { @@ -132,9 +136,15 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { // catch here b/c atomIterator is an AbstractIterator; hasNext reads the value if (e.getCause() instanceof IOException) +{ +if (sstable != null) +sstable.markSuspect(); throw new CorruptSSTableException((IOException)e.getCause(), filename); +} else +{ throw e; +} } } @@ -181,22 +191,39 @@ public class SSTableIdentityIterator implements ComparableSSTableIdentityIterat { ColumnFamily cf = columnFamily.cloneMeShallow(containerFactory, false); // since we already read column count, just pass that value and continue deserialization -IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); -while (iter.hasNext()) -cf.addAtom(iter.next()); +try +{ +IteratorOnDiskAtom iter = cf.metadata().getOnDiskIterator(in, columnCount, flag, expireBefore, dataVersion); +while (iter.hasNext()) +cf.addAtom(iter.next()); -if (validateColumns) +if (validateColumns) +{ +try +{ +cf.metadata().validateColumns(cf); +} +catch (MarshalException e) +{ +throw new RuntimeException(Error validating row + key, e); +} +} +return cf; +} +catch (IOError e) { -try +// catch here b/c atomIterator is an AbstractIterator; hasNext reads the value +if (e.getCause() instanceof IOException) { -cf.metadata().validateColumns(cf); +if (sstable != null) +
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569098#comment-14569098 ] Philip Thompson commented on CASSANDRA-9527: I can confirm this is happening on 2.0.15. [~blerer], should this be possible? If not, then it's just the error message that's wrong. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9530) SSTable corruption can trigger OOM
Sylvain Lebresne created CASSANDRA-9530: --- Summary: SSTable corruption can trigger OOM Key: CASSANDRA-9530 URL: https://issues.apache.org/jira/browse/CASSANDRA-9530 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Priority: Minor If a sstable is corrupted so that the length of a given is bogus, we'll still happily try to allocate a buffer of that bogus size to read the value, which can easily lead to an OOM. We should probably protect against this. In practice, a given value can be so big since it's limited by the protocol frame size in the first place. Maybe we could add a max_value_size_in_mb setting and we'd considered a sstable corrupted if it was containing a value bigger than that. I'll note that this ticket would be a good occasion to improve {{BlacklistingCompactionsTest}}. Typically, it currently generate empty values which makes it pretty much impossible to get the problem described here. And as described in CASSANDRA-9478, it also doesn't test properly for thing like early opening of compaction results. We could try to randomize as much of the parameters of this test as possible to make it more likely to catch any type of corruption that could happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/5] cassandra git commit: Clean up gossiper logic for old versions
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 6f93bd1f6 - 4cfb17f48 refs/heads/trunk ae2daea52 - e27e97413 Clean up gossiper logic for old versions Patch by brandonwilliams, reviewed by aleksey for CASSANDRA-9370 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/23d7a558 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/23d7a558 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/23d7a558 Branch: refs/heads/cassandra-2.2 Commit: 23d7a558ee824a22d382e1b645814638a2cd2985 Parents: 94a68a1 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:42:28 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:42:28 2015 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/gms/Gossiper.java | 16 -- .../cassandra/service/StorageService.java | 54 +--- 3 files changed, 26 insertions(+), 45 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 165f76b..41f5cfb 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411) http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/gms/Gossiper.java -- diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java b/src/java/org/apache/cassandra/gms/Gossiper.java index 0c449ec..e131da6 100644 --- a/src/java/org/apache/cassandra/gms/Gossiper.java +++ b/src/java/org/apache/cassandra/gms/Gossiper.java @@ -798,24 +798,8 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean return endpointStateMap.entrySet(); } -public boolean usesHostId(InetAddress endpoint) -{ -if (MessagingService.instance().knowsVersion(endpoint)) -return true; -else if (getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.NET_VERSION) != null) -return true; -return false; -} - -public boolean usesVnodes(InetAddress endpoint) -{ -return usesHostId(endpoint) getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.TOKENS) != null; -} - public UUID getHostId(InetAddress endpoint) { -if (!usesHostId(endpoint)) -throw new RuntimeException(Host + endpoint + does not use new-style tokens!); return UUID.fromString(getEndpointStateForEndpoint(endpoint).getApplicationState(ApplicationState.HOST_ID).value); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/23d7a558/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index bfbf1a8..8afb93f 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -1825,8 +1825,7 @@ public class StorageService extends NotificationBroadcasterSupport implements IE tokenMetadata.addBootstrapTokens(tokens, endpoint); PendingRangeCalculatorService.instance.update(); -if (Gossiper.instance.usesHostId(endpoint)) -tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint), endpoint); +tokenMetadata.updateHostId(Gossiper.instance.getHostId(endpoint), endpoint); } /** @@ -1855,39 +1854,36 @@ public class StorageService extends NotificationBroadcasterSupport implements IE updatePeerInfo(endpoint); // Order Matters, TM.updateHostID() should be called before TM.updateNormalToken(), (see CASSANDRA-4300). -if (Gossiper.instance.usesHostId(endpoint)) +UUID hostId = Gossiper.instance.getHostId(endpoint); +InetAddress existing = tokenMetadata.getEndpointForHostId(hostId); +if (replacing Gossiper.instance.getEndpointStateForEndpoint(DatabaseDescriptor.getReplaceAddress()) != null (hostId.equals(Gossiper.instance.getHostId(DatabaseDescriptor.getReplaceAddress() +logger.warn(Not updating token metadata for {} because I am replacing it, endpoint); +else { -UUID hostId = Gossiper.instance.getHostId(endpoint); -InetAddress existing =
[4/5] cassandra git commit: Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2
Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2 Conflicts: CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4cfb17f4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4cfb17f4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4cfb17f4 Branch: refs/heads/cassandra-2.2 Commit: 4cfb17f48ea88465b39e7eb762985cda172e1fb3 Parents: 23d7a55 6f93bd1 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:43:15 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:43:15 2015 -0500 -- CHANGES.txt | 2 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../io/sstable/format/SSTableReader.java| 57 +++ .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- .../cassandra/transport/MessagePayloadTest.java | 73 10 files changed, 177 insertions(+), 73 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4cfb17f4/CHANGES.txt -- diff --cc CHANGES.txt index 41f5cfb,db94c76..42846d7 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411)
[3/5] cassandra git commit: Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2
Merge branch 'cassandra-2.2' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-2.2 Conflicts: CHANGES.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4cfb17f4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4cfb17f4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4cfb17f4 Branch: refs/heads/trunk Commit: 4cfb17f48ea88465b39e7eb762985cda172e1fb3 Parents: 23d7a55 6f93bd1 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Jun 2 08:43:15 2015 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Jun 2 08:43:15 2015 -0500 -- CHANGES.txt | 2 + .../CustomPayloadMirroringQueryHandler.java | 18 +++-- .../org/apache/cassandra/cql3/QueryHandler.java | 20 -- .../apache/cassandra/cql3/QueryProcessor.java | 22 -- .../io/sstable/SSTableIdentityIterator.java | 10 +++ .../io/sstable/format/SSTableReader.java| 57 +++ .../org/apache/cassandra/transport/CBUtil.java | 18 ++--- .../org/apache/cassandra/transport/Message.java | 13 ++-- .../compaction/BlacklistingCompactionsTest.java | 17 +++-- .../cassandra/transport/MessagePayloadTest.java | 73 10 files changed, 177 insertions(+), 73 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4cfb17f4/CHANGES.txt -- diff --cc CHANGES.txt index 41f5cfb,db94c76..42846d7 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ 2.2 + * Clean up gossiper logic for old versions (CASSANDRA-9370) + * Fix custom payload coding/decoding to match the spec (CASSANDRA-9515) * ant test-all results incomplete when parsed (CASSANDRA-9463) * Disallow frozen types in function arguments and return types for clarity (CASSANDRA-9411)
[jira] [Created] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
Benjamin Lerer created CASSANDRA-9531: - Summary: NullPointerException logged when running MessagePayloadTest Key: CASSANDRA-9531 URL: https://issues.apache.org/jira/browse/CASSANDRA-9531 Project: Cassandra Issue Type: Bug Reporter: Benjamin Lerer Priority: Minor When I run the {{MessagePayloadTest}} I have the following error in my log output: {code} juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache processPendingNotifications AVERTISSEMENT: Exception thrown by removal listener java.lang.NullPointerException at org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260) at org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105) at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954) at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457) at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433) at com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104) at com.google.common.cache.LocalCache.remove(LocalCache.java:4170) at com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762) at org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182) at org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77) at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285) at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164) at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89) at org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45) at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} The problem is due to the fact that when {{getTotalBufferSize}} is called the segment has already been desallocated. The reason being that the segment has been put twice in the queue of the {{CacheBucket}}. I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2 calls to the {{close}} method of {{BigTableScanner}}. One here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207) {code} and the other here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9514) TrackerTest.testAddSSTables fails in 2.2 and trunk
[ https://issues.apache.org/jira/browse/CASSANDRA-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569287#comment-14569287 ] Michael Shuler commented on CASSANDRA-9514: --- http://cassci.datastax.com/job/trunk_utest/229/testReport/org.apache.cassandra.db.lifecycle/TrackerTest/testAddInitialSSTables/ {noformat} junit.framework.AssertionFailedError at org.apache.cassandra.utils.concurrent.Ref$State.assertNotReleased(Ref.java:157) at org.apache.cassandra.utils.concurrent.Ref.ref(Ref.java:113) at org.apache.cassandra.io.sstable.format.SSTableReader$GlobalTidy.get(SSTableReader.java:2111) at org.apache.cassandra.io.sstable.format.SSTableReader$DescriptorTypeTidy.init(SSTableReader.java:1980) at org.apache.cassandra.io.sstable.format.SSTableReader$DescriptorTypeTidy.get(SSTableReader.java:2017) at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.setup(SSTableReader.java:1897) at org.apache.cassandra.io.sstable.format.SSTableReader.setup(SSTableReader.java:1842) at org.apache.cassandra.io.sstable.format.SSTableReader.internalOpen(SSTableReader.java:530) at org.apache.cassandra.MockSchema.sstable(MockSchema.java:128) at org.apache.cassandra.MockSchema.sstable(MockSchema.java:92) at org.apache.cassandra.MockSchema.sstable(MockSchema.java:87) at org.apache.cassandra.db.lifecycle.TrackerTest.testAddInitialSSTables(TrackerTest.java:146) {noformat} This was on COMMIT = 525579d which looks to be a commit or 2 after this fix was committed - may re-open this.. TrackerTest.testAddSSTables fails in 2.2 and trunk -- Key: CASSANDRA-9514 URL: https://issues.apache.org/jira/browse/CASSANDRA-9514 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Labels: test-failure Fix For: 3.x, 2.2.x Attachments: system.log.gz, system.log.gz Error: {{expected:1 but was:2}} http://cassci.datastax.com/job/cassandra-2.2_testall/33/testReport/org.apache.cassandra.db.lifecycle/TrackerTest/testTryModify/ http://cassci.datastax.com/job/trunk_testall/125/testReport/org.apache.cassandra.db.lifecycle/TrackerTest/testDropSSTables/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.2' into trunk
Merge branch 'cassandra-2.2' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bde7615c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bde7615c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bde7615c Branch: refs/heads/trunk Commit: bde7615c2f8f8d8f6a8a9be4ca01c02fc9e03e54 Parents: e27e974 b2e4d10 Author: Marcus Eriksson marc...@apache.org Authored: Tue Jun 2 17:37:23 2015 +0200 Committer: Marcus Eriksson marc...@apache.org Committed: Tue Jun 2 17:37:23 2015 +0200 -- src/java/org/apache/cassandra/service/StorageService.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bde7615c/src/java/org/apache/cassandra/service/StorageService.java --
cassandra git commit: Make it possible to run dc local repairs
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 4cfb17f48 - b2e4d1015 Make it possible to run dc local repairs Patch by marcuse; reviewed by yukim for CASSANDRA-9521 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b2e4d101 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b2e4d101 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b2e4d101 Branch: refs/heads/cassandra-2.2 Commit: b2e4d10155afa3618ba1a9066c6ffc13489c73d8 Parents: 4cfb17f Author: Marcus Eriksson marc...@apache.org Authored: Tue Jun 2 17:36:42 2015 +0200 Committer: Marcus Eriksson marc...@apache.org Committed: Tue Jun 2 17:36:42 2015 +0200 -- src/java/org/apache/cassandra/service/StorageService.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2e4d101/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 8afb93f..4fdb7b7 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2966,7 +2966,7 @@ public class StorageService extends NotificationBroadcasterSupport implements IE private FutureTaskObject createRepairTask(final int cmd, final String keyspace, final RepairOption options) { -if (!options.getDataCenters().isEmpty() options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter())) +if (!options.getDataCenters().isEmpty() !options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter())) { throw new IllegalArgumentException(the local data center must be part of the repair); }
[1/2] cassandra git commit: Make it possible to run dc local repairs
Repository: cassandra Updated Branches: refs/heads/trunk e27e97413 - bde7615c2 Make it possible to run dc local repairs Patch by marcuse; reviewed by yukim for CASSANDRA-9521 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b2e4d101 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b2e4d101 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b2e4d101 Branch: refs/heads/trunk Commit: b2e4d10155afa3618ba1a9066c6ffc13489c73d8 Parents: 4cfb17f Author: Marcus Eriksson marc...@apache.org Authored: Tue Jun 2 17:36:42 2015 +0200 Committer: Marcus Eriksson marc...@apache.org Committed: Tue Jun 2 17:36:42 2015 +0200 -- src/java/org/apache/cassandra/service/StorageService.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b2e4d101/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 8afb93f..4fdb7b7 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2966,7 +2966,7 @@ public class StorageService extends NotificationBroadcasterSupport implements IE private FutureTaskObject createRepairTask(final int cmd, final String keyspace, final RepairOption options) { -if (!options.getDataCenters().isEmpty() options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter())) +if (!options.getDataCenters().isEmpty() !options.getDataCenters().contains(DatabaseDescriptor.getLocalDataCenter())) { throw new IllegalArgumentException(the local data center must be part of the repair); }
[jira] [Commented] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569296#comment-14569296 ] Benedict commented on CASSANDRA-8897: - Thanks [~stef1927]! I've pushed one last round of suggestions. If you can yay/nay them, and make any comments you feel necessary, I'll then see about committing. They're more involved than I'd first intended, so please don't shy away from any criticism you consider warranted. To summarise, as far as basic functionality is concerned, I've: # modified the chunk queue logic to always be a prefix of non-null items and to always put the new chunk at the end, so it is always tried last # implemented the immediate freeing of a chunk that is owned by the thread that is freeing (to keep retained memory to a minimum) # merged the CAS to perform the recycle with the update that sets it to fully free # some minor renaming/refactoring, such as separating the adoption of a new Chunk from the allocation of a buffer from it Then, I've fixed a bug. This bug crept in due to the mix of suggestions I made to you, and how they combined in this instance. The bug is a pretty subtle and rare concurrency failure, wherein a thread could be late recycling a buffer that has already been recycled and is concurrently being re-adopted by another thread, immediately making it unavailable for allocation by that other thread. In most situations this would end up simply occupying some space in the owner's queue until it got evicted, but it would be possible for it to be reallocated to another thread while still avaiable and owned by the first, and trigger assertion failures. In order to fix this bug, I've rolled back the reuse of the same Chunk instance; there are other solutions, but they are likely not worth the trouble. In order to _exercise_ this bug, I've revamped the long test. The new version attempts to increase the level of direct interleaving of concurrent operations. Sleeps in particular mitigate this kind of behaviour, so they are avoided. Occasionally we yield, if no safe progress can be made by a test thread. There are fewer threads, and these each reduce the concurrent operations they participate in that are not directly incurred in the BufferPool. To summarise these changes: # Utilises DynamicList from stress, that is moved into the main package (and split into LockedDynamicList and DynamicList), and permits us to maintain a per-thread collection of buffers to validate # Constrains the buffer pool's memory so that it can see high churn and reuse of the same chunks # Ensures the total amount of memory in use is at or around the size of the buffer pool # Ensures that all chunks are recycled periodically (so that none are leaked/lost) #* This involved introducing some debug logic into the BufferPool path, to track the recycling of chunks; this isn't super pretty, but is very helpful to have confidence of the behaviour # Randomly frees buffers from its local collection, either itself or via a neighbour # Each thread only frees buffers from one of its neighbours, so that the message passing between them can be kept extremely cheap (reducing cost for none BufferPool operations) # Amortizes the cost of System.nanoTime(), reducing competition within it (again, non-BufferPool concurrent op) # Introduces two new threads, whose sole job is to churn chunks at a high rate. This is to most directly increase the incidence of the bug. # Randomly burst allocates the entire pool contents I've also slightly modified your version, and uploaded it [here|https://github.com/belliottsmith/cassandra/tree/8897-testimprovements] with the new long test. This change was just to further increase the likelihood of exercising this bug, by hitting a failure-to-allocate in the thread that has its freeSlots ripped away from it. Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.x Attachments: 8897.yaml, 9240_test_results.txt, snapshot-1431582436640-cpu-backtraces.png, snapshot-1431582436640-cpu-calltree-compression-8897.nps, snapshot-1431582436640-cpu-calltree-compression-trunk.nps After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9531) NullPointerException logged when running MessagePayloadTest
[ https://issues.apache.org/jira/browse/CASSANDRA-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569298#comment-14569298 ] T Jake Luciani commented on CASSANDRA-9531: --- Branch here https://github.com/tjake/cassandra/tree/9531 I wasn't able to reproduce the issue. But the patch makes the scanner.close call idempotent. Can you verify [~blerer]? NullPointerException logged when running MessagePayloadTest --- Key: CASSANDRA-9531 URL: https://issues.apache.org/jira/browse/CASSANDRA-9531 Project: Cassandra Issue Type: Bug Reporter: Benjamin Lerer Assignee: T Jake Luciani Priority: Minor Fix For: 2.2.0 rc1 When I run the {{MessagePayloadTest}} I have the following error in my log output: {code} juin 02, 2015 3:27:31 PM com.google.common.cache.LocalCache processPendingNotifications AVERTISSEMENT: Exception thrown by removal listener java.lang.NullPointerException at org.apache.cassandra.io.compress.CompressedRandomAccessReader.getTotalBufferSize(CompressedRandomAccessReader.java:260) at org.apache.cassandra.service.FileCacheService$2.onRemoval(FileCacheService.java:105) at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1954) at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3457) at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3433) at com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3104) at com.google.common.cache.LocalCache.remove(LocalCache.java:4170) at com.google.common.cache.LocalCache$LocalManualCache.invalidate(LocalCache.java:4762) at org.apache.cassandra.service.FileCacheService.invalidate(FileCacheService.java:182) at org.apache.cassandra.io.util.PoolingSegmentedFile$Cleanup.tidy(PoolingSegmentedFile.java:53) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup.tidy(CompressedPoolingSegmentedFile.java:77) at org.apache.cassandra.utils.concurrent.Ref$GlobalState.release(Ref.java:285) at org.apache.cassandra.utils.concurrent.Ref$State.ensureReleased(Ref.java:164) at org.apache.cassandra.utils.concurrent.Ref.ensureReleased(Ref.java:89) at org.apache.cassandra.utils.concurrent.SharedCloseableImpl.close(SharedCloseableImpl.java:45) at org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier$1.run(SSTableReader.java:1937) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} The problem is due to the fact that when {{getTotalBufferSize}} is called the segment has already been desallocated. The reason being that the segment has been put twice in the queue of the {{CacheBucket}}. I tracked down the problem to {{CompactionTask.runMayThrow}} which trigger 2 calls to the {{close}} method of {{BigTableScanner}}. One here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.db.compaction.AbstractCompactionStrategy$ScannerList.close(AbstractCompactionStrategy.java:333) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:207) {code} and the other here: {code} at org.apache.cassandra.io.sstable.format.big.BigTableScanner.close(BigTableScanner.java:197) at org.apache.cassandra.utils.MergeIterator.close(MergeIterator.java:62) at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:206) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569313#comment-14569313 ] Michael Shuler commented on CASSANDRA-8656: --- 30x over https://github.com/stef1927/cassandra/commit/25ecbff with no failures. :) long-test LongLeveledCompactionStrategyTest flaps in 2.0 Key: CASSANDRA-8656 URL: https://issues.apache.org/jira/browse/CASSANDRA-8656 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Priority: Minor Labels: test-failure Fix For: 2.0.x Attachments: system.log LongLeveledCompactionStrategyTest periodically fails with: {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.412 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97) [junit] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409) [junit] Caused by: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) [junit] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) [junit] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [junit] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) [junit] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87) [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED {noformat} I would guess the failure is 10-20% of the time, looping over the test repeatedly. On the 2.1 branch, the failure is different, so perhaps this could also be updated. {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.04 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy
[jira] [Commented] (CASSANDRA-9527) Cannot create secondary index on a table WITH COMPACT STORAGE
[ https://issues.apache.org/jira/browse/CASSANDRA-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569314#comment-14569314 ] Benjamin Lerer commented on CASSANDRA-9527: --- Sorry, I was wrong. We support secondary indices only on the compact tables that does not have clustering columns. Which means that we should change the error message for this specific case. Cannot create secondary index on a table WITH COMPACT STORAGE - Key: CASSANDRA-9527 URL: https://issues.apache.org/jira/browse/CASSANDRA-9527 Project: Cassandra Issue Type: Bug Components: Core Environment: cassandra v2.0.13 Reporter: fuggy_yama Assignee: Benjamin Lerer Priority: Minor Fix For: 2.0.x In CASSANDRA-8156 it is said that secondary indexes are not allowed on clustering columns in COMPACT tables. However, I found that it is not possible to create a secondary index on the value column in a COMPACT table: CREATE TABLE t ( a INT, b INT, c INT, PRIMARY KEY (a, b) ) WITH COMPACT STORAGE; CREATE INDEX ON t (c); *Bad Request: Secondary indexes are not supported on PRIMARY KEY columns in COMPACT STORAGE tables* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569299#comment-14569299 ] Benedict commented on CASSANDRA-8897: - Thanks [~stef1927]! I've pushed one last round of suggestions. If you can yay/nay them, and make any comments you feel necessary, I'll then see about committing. They're more involved than I'd first intended, so please don't shy away from any criticism you consider warranted. To summarise, as far as basic functionality is concerned, I've: # modified the chunk queue logic to always be a prefix of non-null items and to always put the new chunk at the end, so it is always tried last # implemented the immediate freeing of a chunk that is owned by the thread that is freeing (to keep retained memory to a minimum) # merged the CAS to perform the recycle with the update that sets it to fully free # some minor renaming/refactoring, such as separating the adoption of a new Chunk from the allocation of a buffer from it Then, I've fixed a bug. This bug crept in due to the mix of suggestions I made to you, and how they combined in this instance. The bug is a pretty subtle and rare concurrency failure, wherein a thread could be late recycling a buffer that has already been recycled and is concurrently being re-adopted by another thread, immediately making it unavailable for allocation by that other thread. In most situations this would end up simply occupying some space in the owner's queue until it got evicted, but it would be possible for it to be reallocated to another thread while still avaiable and owned by the first, and trigger assertion failures. In order to fix this bug, I've rolled back the reuse of the same Chunk instance; there are other solutions, but they are likely not worth the trouble. In order to _exercise_ this bug, I've revamped the long test. The new version attempts to increase the level of direct interleaving of concurrent operations. Sleeps in particular mitigate this kind of behaviour, so they are avoided. Occasionally we yield, if no safe progress can be made by a test thread. There are fewer threads, and these each reduce the concurrent operations they participate in that are not directly incurred in the BufferPool. To summarise these changes: # Utilises DynamicList from stress, that is moved into the main package (and split into LockedDynamicList and DynamicList), and permits us to maintain a per-thread collection of buffers to validate # Constrains the buffer pool's memory so that it can see high churn and reuse of the same chunks # Ensures the total amount of memory in use is at or around the size of the buffer pool # Ensures that all chunks are recycled periodically (so that none are leaked/lost) #* This involved introducing some debug logic into the BufferPool path, to track the recycling of chunks; this isn't super pretty, but is very helpful to have confidence of the behaviour # Randomly frees buffers from its local collection, either itself or via a neighbour # Each thread only frees buffers from one of its neighbours, so that the message passing between them can be kept extremely cheap (reducing cost for none BufferPool operations) # Amortizes the cost of System.nanoTime(), reducing competition within it (again, non-BufferPool concurrent op) # Introduces two new threads, whose sole job is to churn chunks at a high rate. This is to most directly increase the incidence of the bug. # Randomly burst allocates the entire pool contents I've also slightly modified your version, and uploaded it [here|https://github.com/belliottsmith/cassandra/tree/8897-testimprovements] with the new long test. This change was just to further increase the likelihood of exercising this bug, by hitting a failure-to-allocate in the thread that has its freeSlots ripped away from it. Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.x Attachments: 8897.yaml, 9240_test_results.txt, snapshot-1431582436640-cpu-backtraces.png, snapshot-1431582436640-cpu-calltree-compression-8897.nps, snapshot-1431582436640-cpu-calltree-compression-trunk.nps After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8805) runWithCompactionsDisabled only cancels compactions, which is not the only source of markCompacted
[ https://issues.apache.org/jira/browse/CASSANDRA-8805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569384#comment-14569384 ] Benedict commented on CASSANDRA-8805: - The rollback logic in the event of CIE is, unfortunately, likely to break things. Once a reader replaces another, it needs to be published or itself replaced. So we should either make visible the newly sampled readers (excluding the ones we haven't completed), or we should construct a new instance with the old summaries that are publised. Since the operation can also be very expensive, we should probably extract the resampling into a method that can itself check for the {{isStopRequested}}, otherwise it could take a long time to acquiesce. runWithCompactionsDisabled only cancels compactions, which is not the only source of markCompacted -- Key: CASSANDRA-8805 URL: https://issues.apache.org/jira/browse/CASSANDRA-8805 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benedict Assignee: Carl Yeksigian Fix For: 2.1.x Attachments: 8805-2.1.txt Operations like repair that may operate over all sstables cancel compactions before beginning, and fail if there are any files marked compacting after doing so. Redistribution of index summaries is not a compaction, so is not cancelled by this action, but does mark sstables as compacting, so such an action will fail to initiate if there is an index summary redistribution in progress. It seems that IndexSummaryManager needs to register itself as interruptible along with compactions (AFAICT no other actions that may markCompacting are not themselves compactions). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-8897: Comment: was deleted (was: Thanks [~stef1927]! I've pushed one last round of suggestions. If you can yay/nay them, and make any comments you feel necessary, I'll then see about committing. They're more involved than I'd first intended, so please don't shy away from any criticism you consider warranted. To summarise, as far as basic functionality is concerned, I've: # modified the chunk queue logic to always be a prefix of non-null items and to always put the new chunk at the end, so it is always tried last # implemented the immediate freeing of a chunk that is owned by the thread that is freeing (to keep retained memory to a minimum) # merged the CAS to perform the recycle with the update that sets it to fully free # some minor renaming/refactoring, such as separating the adoption of a new Chunk from the allocation of a buffer from it Then, I've fixed a bug. This bug crept in due to the mix of suggestions I made to you, and how they combined in this instance. The bug is a pretty subtle and rare concurrency failure, wherein a thread could be late recycling a buffer that has already been recycled and is concurrently being re-adopted by another thread, immediately making it unavailable for allocation by that other thread. In most situations this would end up simply occupying some space in the owner's queue until it got evicted, but it would be possible for it to be reallocated to another thread while still avaiable and owned by the first, and trigger assertion failures. In order to fix this bug, I've rolled back the reuse of the same Chunk instance; there are other solutions, but they are likely not worth the trouble. In order to _exercise_ this bug, I've revamped the long test. The new version attempts to increase the level of direct interleaving of concurrent operations. Sleeps in particular mitigate this kind of behaviour, so they are avoided. Occasionally we yield, if no safe progress can be made by a test thread. There are fewer threads, and these each reduce the concurrent operations they participate in that are not directly incurred in the BufferPool. To summarise these changes: # Utilises DynamicList from stress, that is moved into the main package (and split into LockedDynamicList and DynamicList), and permits us to maintain a per-thread collection of buffers to validate # Constrains the buffer pool's memory so that it can see high churn and reuse of the same chunks # Ensures the total amount of memory in use is at or around the size of the buffer pool # Ensures that all chunks are recycled periodically (so that none are leaked/lost) #* This involved introducing some debug logic into the BufferPool path, to track the recycling of chunks; this isn't super pretty, but is very helpful to have confidence of the behaviour # Randomly frees buffers from its local collection, either itself or via a neighbour # Each thread only frees buffers from one of its neighbours, so that the message passing between them can be kept extremely cheap (reducing cost for none BufferPool operations) # Amortizes the cost of System.nanoTime(), reducing competition within it (again, non-BufferPool concurrent op) # Introduces two new threads, whose sole job is to churn chunks at a high rate. This is to most directly increase the incidence of the bug. # Randomly burst allocates the entire pool contents I've also slightly modified your version, and uploaded it [here|https://github.com/belliottsmith/cassandra/tree/8897-testimprovements] with the new long test. This change was just to further increase the likelihood of exercising this bug, by hitting a failure-to-allocate in the thread that has its freeSlots ripped away from it.) Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.x Attachments: 8897.yaml, 9240_test_results.txt, snapshot-1431582436640-cpu-backtraces.png, snapshot-1431582436640-cpu-calltree-compression-8897.nps, snapshot-1431582436640-cpu-calltree-compression-trunk.nps After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-8656: -- Comment: was deleted (was: 30x over https://github.com/stef1927/cassandra/commit/25ecbff with no failures. :)) long-test LongLeveledCompactionStrategyTest flaps in 2.0 Key: CASSANDRA-8656 URL: https://issues.apache.org/jira/browse/CASSANDRA-8656 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Priority: Minor Labels: test-failure Fix For: 2.0.x Attachments: system.log LongLeveledCompactionStrategyTest periodically fails with: {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.412 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97) [junit] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409) [junit] Caused by: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) [junit] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) [junit] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [junit] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) [junit] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87) [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED {noformat} I would guess the failure is 10-20% of the time, looping over the test repeatedly. On the 2.1 branch, the failure is different, so perhaps this could also be updated. {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.04 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy cannot be cast to
[jira] [Commented] (CASSANDRA-8656) long-test LongLeveledCompactionStrategyTest flaps in 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569301#comment-14569301 ] Michael Shuler commented on CASSANDRA-8656: --- 30x over https://github.com/stef1927/cassandra/commit/25ecbff with no failures. :) long-test LongLeveledCompactionStrategyTest flaps in 2.0 Key: CASSANDRA-8656 URL: https://issues.apache.org/jira/browse/CASSANDRA-8656 Project: Cassandra Issue Type: Test Components: Tests Reporter: Michael Shuler Assignee: Stefania Priority: Minor Labels: test-failure Fix For: 2.0.x Attachments: system.log LongLeveledCompactionStrategyTest periodically fails with: {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 54.412 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:402) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest.testParallelLeveledCompaction(LongLeveledCompactionStrategyTest.java:97) [junit] Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit] at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit] at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409) [junit] Caused by: java.lang.RuntimeException: Last written key DecoratedKey(3133, 3133) = current key DecoratedKey(313236, 313236) writing into build/test/cassandra/data/Keyspace1/StandardLeveled/Keyspace1-StandardLeveled-tmp-jb-304-Data.db [junit] at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143) [junit] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166) [junit] at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167) [junit] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [junit] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) [junit] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) [junit] at org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest$1.run(LongLeveledCompactionStrategyTest.java:87) [junit] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit] at java.lang.Thread.run(Thread.java:745) [junit] [junit] [junit] Test org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest FAILED {noformat} I would guess the failure is 10-20% of the time, looping over the test repeatedly. On the 2.1 branch, the failure is different, so perhaps this could also be updated. {noformat} [junit] Testsuite: org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.04 sec [junit] [junit] Testcase: testParallelLeveledCompaction(org.apache.cassandra.db.compaction.LongLeveledCompactionStrategyTest): Caused an ERROR [junit] org.apache.cassandra.db.compaction.WrappingCompactionStrategy
[jira] [Created] (CASSANDRA-9532) Provide access to select statement's real column definitions
mck created CASSANDRA-9532: -- Summary: Provide access to select statement's real column definitions Key: CASSANDRA-9532 URL: https://issues.apache.org/jira/browse/CASSANDRA-9532 Project: Cassandra Issue Type: Improvement Reporter: mck Assignee: mck Currently there is no way to get access to the real ColumnDefinitions being used in a SelectStatement. This information is there in {{selectStatement.selection.columns}} but is private. Giving public access would make it possible for third-party implementations of a {{QueryHandler}} to work accurately with the real columns being queried and not have to work-around column aliases (or when the rawSelectors don't map directly to ColumnDefinitions, eg in Selection.fromSelectors(..), like functions), which is what one has to do today with going through ResultSet.metadata.names. This issue provides a very minimal patch to provide access to the already final and immutable fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8915) Improve MergeIterator performance
[ https://issues.apache.org/jira/browse/CASSANDRA-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14569392#comment-14569392 ] Branimir Lambov commented on CASSANDRA-8915: I wasn't completely sure of the correctness of the flagging approach and I wrote the following proof to make sure I'm not missing something: {panel} In the discussion below ρ(N) stands for parent of N and σ(N) stands for the 'equalParent' predicate, i.e. σ(N) ↔ ρ(N) = N. We define ep-heap leading to M to be a binary tree of nodes where each node N satisfies ρ(N) ≤ N σ(N) ↔ ρ(N) = N and the root has M as its parent. Full ep-heaps are ep-heaps leading to top, where 'top' is assumed to be a special value strictly smaller than any other (thus σ(R) is always false for the root of a full ep-heap). For simplicity we will take all heaps to end at a special node called 'bottom' whose value is assumed to be greater than any other value. The critical step here is adding a new value C M to combine two existing well-formed ep-heaps P and R leading to the same M into a bigger single well-formed ep-heap leading to M. Without loss of generality we can take P ≤ R (otherwise we can swap the places of P and R in the discussion below). We distinguish three cases: # C P (necessarily ¬σ(P)): We place C at the top by setting ρ(P), ρ(R) := C, ρ(C) := M and leaving σ(P), σ(R) := false (Note that σ(R) cannot be true as that contradicts with C M). σ(C) := false. The ep-heap rooted at C is well-formed leading to M. # C = P (necessarily ¬σ(P)): We place C at the top, setting ρ(P), ρ(R) := C, ρ(C) := M, σ(P) := true and σ(R) := (P = R). σ(C) := false. The heap rooted at C is well-formed as C = P σ(P) as well as C = P ≤ R C = R if and only if σ(R). # C P (σ(P) can be true or false): Set ρ(R) := P. If ¬σ(P), set σ(R) := (P = R), otherwise leave σ(R) unchanged. Use the algorithm recursively to turn the two ep-subheaps leading to P at its children and the new value C P into an ep-heap leading to P. Let its root be Q. The necessary conditions for an element of an ep-heap hold for Q. They are also true for R as P ≤ R and either σ(P) in which case P = M and σ(R) if and only if R = M = P, or ¬σ(P) and σ(R) is explicitly set to be true if and only if P = R. We also have ρ(P) = M as well as P = M if and only if σ(P), thus the constructed ep-heap rooted at P leads to M. The recursion always ends for finite heaps as we must reach the first case due to the ordering of bottom. For performance the order of comparing elements and identifying the case to use can be changed to sequentially checking: - σ(P): apply pt. 3 for P and return (P must be ≤ R). No update of σ(P)/σ(R) necessary. - σ(R): apply pt. 3 for R and return (R must now be P). No update of σ(P)/σ(R) necessary. - compare P and R. - If R P, perform the next steps with P and R swapped. - compare C and P. - if C P apply pt. 1. - if C = P apply pt. 2. - if C P apply pt. 3. Once an ep-heap with root R is formed, it trivially follows that for each element E in the heap E = R if and only if σ is true for E and the chain of parents of E leading to but not including the root. To consume the equal elements we can thus walk the sub-tree formed by the children which have σ set. To advance the elements equal to the root, we can: - Process the elements in the deepest level of the σ sub-tree in any order. For each of them: -- Advance the element. Let the new value be C. -- The children P, R of the element must have ¬σ(P), ¬σ(R). Overwrite ρ(P) and ρ(R) with top*. P and R now form ep-heaps leading to top. -- Use the algorithm to combine P, R and C into a new ep-heap leading to top. -- Let its root be Q. Because the heap leads to top we have ¬σ(Q). - Continue with the next deepest level of the hierarchy until the top is reached. \* The algorithm never actually uses the values of the parent function ρ, only the fact that σ is set correctly with respect to that parent. This means this step does not need to be actually performed. {panel} Updated [the branch|https://github.com/apache/cassandra/compare/trunk...blambov:8915-mergeiterator] to use this solution and cleaned the code a bit. Improve MergeIterator performance - Key: CASSANDRA-8915 URL: https://issues.apache.org/jira/browse/CASSANDRA-8915 Project: Cassandra Issue Type: Improvement Reporter: Branimir Lambov Assignee: Branimir Lambov Priority: Minor Labels: compaction, performance Fix For: 3.x The implementation of {{MergeIterator}} uses a priority queue and applies a pair of {{poll}}+{{add}} operations for every item in the resulting sequence. This is quite inefficient as {{poll}} necessarily applies at least {{log N}} comparisons (up to {{2log N}}), and {{add}}