[jira] [Commented] (CASSANDRA-3967) Test fixes for Windows
[ https://issues.apache.org/jira/browse/CASSANDRA-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218019#comment-13218019 ] Sylvain Lebresne commented on CASSANDRA-3967: - The problem is that most tests rely on the behavior of CFS.clearUnsafe to not remove sstables. Typically, in SSTableReaderTest, there is 'persistence' tests that use clearUnsafe followed by a loadNewSSTable to check we correctly reload sstables. CASSANDRA-3970 is another example where we rely on the sstable not being removed. Also ColumnFamilyStoreTest and RecoveryManagerTest. The patch also seem to be breaking a test in DatabaseDescriptorTest (testTransKsMigration that doesn't fail without the patch) but that test doesn't use clearUnsafe so I'm not sure what's going on. > Test fixes for Windows > -- > > Key: CASSANDRA-3967 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3967 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Fix For: 1.1.0 > > Attachments: 3967.txt > > > - SchemaLoader initializes CommitLog, creating empty segments; CleanupHelper > then errors out trying to delete them (because they are still open) > - CFS.clearUnsafe resets the sstable generation but does not remove the > sstables involved, so Windows will error out trying to rename over the > existing ones -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3970) (100% reproducible) JVM crash in streamingTransferTest on Windows
[ https://issues.apache.org/jira/browse/CASSANDRA-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218020#comment-13218020 ] Sylvain Lebresne commented on CASSANDRA-3970: - I believe CASSANDRA-3967 actually create the error by making CFS.clearUnsafe() remove the sstable. The way the test work, it creates a sstable (inserting data the usual way then flushing), and then use clearUnsafe to unreference it. It then use streaming to transfer that now non-referenced sstable to the node (thus the sstable should *not* have be deleted from disk). It's arguably a bit of a hack to allow testing streaming in a single node setting and we could copy the initial sstable to a temporary location before calling clearUnsafe (but since it's not the only thing the patch for CASSANDRA-3967 breaks, let's first see what we do for CASSANDRA-3967). > (100% reproducible) JVM crash in streamingTransferTest on Windows > - > > Key: CASSANDRA-3970 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3970 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Jonathan Ellis >Assignee: Sylvain Lebresne > Fix For: 1.1.0 > > Attachments: hs_err_pid95744.log > > > {noformat} > $ ant test -Dtest.name=StreamingTransferTest > ... > [junit] Testsuite: org.apache.cassandra.streaming.StreamingTransferTest > [junit] # > [junit] # A fatal error has been detected by the Java Runtime Environment: > [junit] # > [junit] # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x6da5ccca, > pid=95744, tid=94924 > [junit] # > [junit] # JRE version: 6.0_27-b07 > [junit] # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode > windows-amd64 compressed oops) > [junit] # Problematic frame: > [junit] # V [jvm.dll+0x1a] > [junit] # > [junit] # An error report file with more information is saved as: > [junit] # c:\Users\Jonathan\projects\cassandra\git\hs_err_pid95744.log > [junit] # > [junit] # If you would like to submit a bug report, please visit: > [junit] # http://java.sun.com/webapps/bugreport/crash.jsp > [junit] # > [junit] Testsuite: org.apache.cassandra.streaming.StreamingTransferTest > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] > [junit] Testcase: > org.apache.cassandra.streaming.StreamingTransferTest:testTransferTable: > Caused an ERROR > [junit] Forked Java VM exited abnormally. Please note the time in the report > does not reflect the time until the VM exit. > [junit] junit.framework.AssertionFailedError: Forked Java VM exited > abnormally. Please note the time in the report does not reflect the time > until the VM exit. > [junit] > [junit] > [junit] Test org.apache.cassandra.streaming.StreamingTransferTest FAILED > (crashed) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[2/2] git commit: Make SS.hasSameReplication safer/more readable
Make SS.hasSameReplication safer/more readable patch by dbrosius; reviewed by slebresne for CASSANDRA-3971 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/24939a58 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/24939a58 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/24939a58 Branch: refs/heads/cassandra-1.1 Commit: 24939a58b381589461677e03c2b19a2035d9c727 Parents: 449e037 Author: Sylvain Lebresne Authored: Tue Feb 28 10:49:29 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 10:49:29 2012 +0100 -- .../apache/cassandra/service/StorageService.java |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/24939a58/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 9ae666c..d50e21e 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2670,10 +2670,11 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; + for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[1/2] git commit: Remove duplicate code
Updated Branches: refs/heads/cassandra-1.1 449e03719 -> d079c8204 Remove duplicate code patch by dbrosius; reviewed by slebresne for CASSANDRA-3969 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d079c820 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d079c820 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d079c820 Branch: refs/heads/cassandra-1.1 Commit: d079c8204e5e187b36b8d0cf66556c60cf33043e Parents: 24939a5 Author: Sylvain Lebresne Authored: Tue Feb 28 11:04:31 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 11:04:31 2012 +0100 -- .../cassandra/service/RangeSliceVerbHandler.java |1 - 1 files changed, 0 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d079c820/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java -- diff --git a/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java b/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java index 76823de..d18509d 100644 --- a/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java +++ b/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java @@ -60,7 +60,6 @@ public class RangeSliceVerbHandler implements IVerbHandler throw new RuntimeException("Cannot service reads while bootstrapping!"); } RangeSliceCommand command = RangeSliceCommand.read(message); -ColumnFamilyStore cfs = Table.open(command.keyspace).getColumnFamilyStore(command.column_family); RangeSliceReply reply = new RangeSliceReply(executeLocally(command)); Message response = reply.getReply(message); if (logger.isDebugEnabled())
[2/3] git commit: Remove duplicate code
Remove duplicate code patch by dbrosius; reviewed by slebresne for CASSANDRA-3969 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d079c820 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d079c820 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d079c820 Branch: refs/heads/trunk Commit: d079c8204e5e187b36b8d0cf66556c60cf33043e Parents: 24939a5 Author: Sylvain Lebresne Authored: Tue Feb 28 11:04:31 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 11:04:31 2012 +0100 -- .../cassandra/service/RangeSliceVerbHandler.java |1 - 1 files changed, 0 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d079c820/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java -- diff --git a/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java b/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java index 76823de..d18509d 100644 --- a/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java +++ b/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java @@ -60,7 +60,6 @@ public class RangeSliceVerbHandler implements IVerbHandler throw new RuntimeException("Cannot service reads while bootstrapping!"); } RangeSliceCommand command = RangeSliceCommand.read(message); -ColumnFamilyStore cfs = Table.open(command.keyspace).getColumnFamilyStore(command.column_family); RangeSliceReply reply = new RangeSliceReply(executeLocally(command)); Message response = reply.getReply(message); if (logger.isDebugEnabled())
[1/3] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches: refs/heads/trunk 1c507e5f9 -> d82814ae4 Merge branch 'cassandra-1.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d82814ae Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d82814ae Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d82814ae Branch: refs/heads/trunk Commit: d82814ae4960cb7966137578f12902c5ea8b279c Parents: 1c507e5 d079c82 Author: Sylvain Lebresne Authored: Tue Feb 28 11:19:49 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 11:19:49 2012 +0100 -- .../cassandra/service/RangeSliceVerbHandler.java |1 - .../apache/cassandra/service/StorageService.java |5 +++-- 2 files changed, 3 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d82814ae/src/java/org/apache/cassandra/service/RangeSliceVerbHandler.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/d82814ae/src/java/org/apache/cassandra/service/StorageService.java --
[3/3] git commit: Make SS.hasSameReplication safer/more readable
Make SS.hasSameReplication safer/more readable patch by dbrosius; reviewed by slebresne for CASSANDRA-3971 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/24939a58 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/24939a58 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/24939a58 Branch: refs/heads/trunk Commit: 24939a58b381589461677e03c2b19a2035d9c727 Parents: 449e037 Author: Sylvain Lebresne Authored: Tue Feb 28 10:49:29 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 10:49:29 2012 +0100 -- .../apache/cassandra/service/StorageService.java |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/24939a58/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 9ae666c..d50e21e 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2670,10 +2670,11 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; + for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[jira] [Updated] (CASSANDRA-3668) Performance of sstableloader is affected in 1.0.x
[ https://issues.apache.org/jira/browse/CASSANDRA-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-3668: Fix Version/s: (was: 1.1.0) 1.1.1 > Performance of sstableloader is affected in 1.0.x > - > > Key: CASSANDRA-3668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3668 > Project: Cassandra > Issue Type: Bug > Components: API >Affects Versions: 1.0.0 >Reporter: Manish Zope >Assignee: Yuki Morishita >Priority: Minor > Labels: streaming > Fix For: 1.1.1 > > Attachments: 0001-Allow-multiple-connection-in-StreamInSession.patch, > 0002-Allow-concurrent-stream-in-StreamOutSession.patch, > 0003-Add-threads-option-to-sstableloader.patch, > 3688-reply_before_closing_writer.txt, sstable-loader performance.txt > > Original Estimate: 48h > Remaining Estimate: 48h > > One of my colleague had reported the bug regarding the degraded performance > of the sstable generator and sstable loader. > ISSUE :- https://issues.apache.org/jira/browse/CASSANDRA-3589 > As stated in above issue generator performance is rectified but performance > of the sstableloader is still an issue. > 3589 is marked as duplicate of 3618.Both issues shows resolved status.But the > problem with sstableloader still exists. > So opening other issue so that sstbleloader problem should not go unnoticed. > FYI : We have tested the generator part with the patch given in 3589.Its > Working fine. > Please let us know if you guys require further inputs from our side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3668) Performance of sstableloader is affected in 1.0.x
[ https://issues.apache.org/jira/browse/CASSANDRA-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218046#comment-13218046 ] Sylvain Lebresne commented on CASSANDRA-3668: - @Yukim Would you mind rebasing? Also, you said getting better throughput with this patch, but could you give a few more details on that (like numbers and what kind of tests you performed). I moved this to 1.1.1. This modify streaming and I'm not confortable doing that on 1.1.0 at this point of the freeze. > Performance of sstableloader is affected in 1.0.x > - > > Key: CASSANDRA-3668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3668 > Project: Cassandra > Issue Type: Bug > Components: API >Affects Versions: 1.0.0 >Reporter: Manish Zope >Assignee: Yuki Morishita >Priority: Minor > Labels: streaming > Fix For: 1.1.1 > > Attachments: 0001-Allow-multiple-connection-in-StreamInSession.patch, > 0002-Allow-concurrent-stream-in-StreamOutSession.patch, > 0003-Add-threads-option-to-sstableloader.patch, > 3688-reply_before_closing_writer.txt, sstable-loader performance.txt > > Original Estimate: 48h > Remaining Estimate: 48h > > One of my colleague had reported the bug regarding the degraded performance > of the sstable generator and sstable loader. > ISSUE :- https://issues.apache.org/jira/browse/CASSANDRA-3589 > As stated in above issue generator performance is rectified but performance > of the sstableloader is still an issue. > 3589 is marked as duplicate of 3618.Both issues shows resolved status.But the > problem with sstableloader still exists. > So opening other issue so that sstbleloader problem should not go unnoticed. > FYI : We have tested the generator part with the patch given in 3589.Its > Working fine. > Please let us know if you guys require further inputs from our side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218057#comment-13218057 ] Sylvain Lebresne commented on CASSANDRA-3442: - Agreed on targeting for 1.2. There is now a branch for it though if that's ready (the patches probably need rebase though). > TTL histogram for sstable metadata > -- > > Key: CASSANDRA-3442 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3442 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jonathan Ellis >Assignee: Yuki Morishita >Priority: Minor > Labels: compaction > Fix For: 1.2 > > Attachments: cassandra-1.1-3442.txt > > > Under size-tiered compaction, you can generate large sstables that compact > infrequently. With expiring columns mixed in, we could waste a lot of space > in this situation. > If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a > single-sstable compaction aginst sstables with over 20% (?) expired data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
HintedHandoff fails to deliver any hints Key: CASSANDRA-3972 URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0 Reporter: Brandon Williams Assignee: Brandon Williams Priority: Blocker Fix For: 1.1.0 Summary says it all. Whether in a memtable or sstable, no hints are delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218143#comment-13218143 ] Brandon Williams commented on CASSANDRA-3972: - At least one problem here is that CFS.removeDeleted is removing everything here (though listing the hints family in the cli works correctly): {code} ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), Integer.MAX_VALUE); {code} At DEBUG you can see the columns collected, but then hintsPage is null. If we simply change this to: {code} ColumnFamily hintsPage = hintStore.getColumnFamily(filter); {code} Then at least some of the hints get sent. > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3769) Allow for comments in a cassandra-cli file
[ https://issues.apache.org/jira/browse/CASSANDRA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218159#comment-13218159 ] Benson Margulies commented on CASSANDRA-3769: - How did my name get onto this? > Allow for comments in a cassandra-cli file > -- > > Key: CASSANDRA-3769 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3769 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Affects Versions: 1.0.6 > Environment: Amazon Linux w/ Apache Cassandra 1.0.6 >Reporter: Andrew Halloran >Priority: Trivial > Labels: cassandra-cli > > I use the cassandra-cli to create schemas, update schemas, and make other > calls to interrogate keyspaces and columns. I load pre-written statements > from files using the -f option, example: "bin/cassandra-cli -host localhost > -port 9160 -f mystatements.txt". It would be useful if I could comment my > statement files with comments similar to how you can comment script and > C++/Java code. > Example contents of "mystatements.txt" file: > update column family users// This is my column which > holds all my user information > with comparator = UTF8Type// My column names are all > strings so I will use UTF8Type > and key_validation_class = UTF8Type // My row key values are also > UTF8Type > and default_validation_class = UTF8Type // Column values will be > UTF8Type > and column_metadata = [ > {column_name: username, validation_class: UTF8Type}, // This > column stores the login name of the user > {column_name: realname, validation_class: UTF8Type}];// This > column stores the real world name of user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3859) Add Progress Reporting to Cassandra OutputFormats
[ https://issues.apache.org/jira/browse/CASSANDRA-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218192#comment-13218192 ] Samarth Gahire commented on CASSANDRA-3859: --- Ok ,So I tested patch for BulkOutputFormat with time out of 30 seconds. 1) First of all I tried it without applying a patch and it was throwing a timeout exception. 2) Then I applied a patch and it worked properly. It means with patch progress reporting is working but it is not reporting progress after every second while loading(can you explain this?). Because same patch throws timeout exception for 10 seconds of time out. > Add Progress Reporting to Cassandra OutputFormats > - > > Key: CASSANDRA-3859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3859 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop, Tools >Affects Versions: 1.1.0 >Reporter: Samarth Gahire >Assignee: Brandon Williams >Priority: Minor > Labels: bulkloader, hadoop, mapreduce, sstableloader > Fix For: 1.1.0 > > Attachments: 0001-add-progress-reporting-to-BOF.txt, > 0002-Add-progress-to-CFOF.txt > > Original Estimate: 48h > Remaining Estimate: 48h > > When we are using the BulkOutputFormat to load the data to cassandra. We > should use the progress reporting to Hadoop Job within Sstable loader because > while loading the data for particular task if streaming is taking more time > and progress is not reported to Job it may kill the task with timeout > exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3859) Add Progress Reporting to Cassandra OutputFormats
[ https://issues.apache.org/jira/browse/CASSANDRA-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218193#comment-13218193 ] Brandon Williams commented on CASSANDRA-3859: - bq. It means with patch progress reporting is working but it is not reporting progress after every second while loading(can you explain this?). Because same patch throws timeout exception for 10 seconds of time out. My guess is 10s is too artificial an amount of time and the JVM is still forking, or warming up, or something. > Add Progress Reporting to Cassandra OutputFormats > - > > Key: CASSANDRA-3859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3859 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop, Tools >Affects Versions: 1.1.0 >Reporter: Samarth Gahire >Assignee: Brandon Williams >Priority: Minor > Labels: bulkloader, hadoop, mapreduce, sstableloader > Fix For: 1.1.0 > > Attachments: 0001-add-progress-reporting-to-BOF.txt, > 0002-Add-progress-to-CFOF.txt > > Original Estimate: 48h > Remaining Estimate: 48h > > When we are using the BulkOutputFormat to load the data to cassandra. We > should use the progress reporting to Hadoop Job within Sstable loader because > while loading the data for particular task if streaming is taking more time > and progress is not reported to Job it may kill the task with timeout > exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of "GettingStarted" by MakiWatanabe
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Cassandra Wiki" for change notification. The "GettingStarted" page has been changed by MakiWatanabe: http://wiki.apache.org/cassandra/GettingStarted?action=diff&rev1=62&rev2=63 Comment: Update nodetool ring sample to 1.0 format/Add short comment about jna * If you are interested in building Cassandra from source, please refer to [[HowToBuild|How to Build]] page. For more details about misc builds, please refer to [[VersionsAndBuilds|Cassandra versions and builds]] page. + + If you plan to run "snapshot" command on Cassandra, it will be better to install jna.jar also. Please refer to [[Operations#Backing_up_data|Backup Data]] section. <> @@ -218, +220 @@ Once everything is configured and the nodes are running, use the `bin/nodetool ring` utility to verify a properly connected cluster. For example: {{{ - eevans@achilles:‾$ bin/nodetool -host 98.139.220.175 -p 7199 ring + eevans@achilles:‾$ bin/nodetool -host 192.168.0.10 -p 7199 ring - Address Status Load Range Ring + Address DC RackStatus State LoadOwnsToken -169048975998562660269742699624378098572 - 98.139.220.175 Up 0.02 GB 14183696824377310051808173385764689249 |<--| - 98.139.169.152 Up 0.4 GB 28356863910078205288614550619314017621 | ^ - 98.139.220.176 Up 0.13 GB 42530828068625072228863933889289238187 |-->| + 127605887595351923798765477786913079296 + 192.168.0.10DC1 r1 Up Normal 17.3 MB 25.00% 0 + 192.168.0.11DC1 r1 Up Normal 17.4 MB 25.00% 42535295865117307932921825928971026432 + 192.168.0.12DC1 r1 Up Normal 37.2 MB 25.00% 85070591730234615865843651857942052864 + 192.168.0.13DC1 r1 Up Normal 24.55 MB25.00% 127605887595351923798765477786913079296 }}} Advanced cluster management is described in [[Operations]].
[jira] [Commented] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218227#comment-13218227 ] Jonathan Ellis commented on CASSANDRA-3972: --- removeDeleted should just drop tombstones. > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218226#comment-13218226 ] Jonathan Ellis commented on CASSANDRA-3442: --- Yes, I'm working on an updated patch. > TTL histogram for sstable metadata > -- > > Key: CASSANDRA-3442 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3442 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jonathan Ellis >Assignee: Yuki Morishita >Priority: Minor > Labels: compaction > Fix For: 1.2 > > Attachments: cassandra-1.1-3442.txt > > > Under size-tiered compaction, you can generate large sstables that compact > infrequently. With expiring columns mixed in, we could waste a lot of space > in this situation. > If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a > single-sstable compaction aginst sstables with over 20% (?) expired data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218248#comment-13218248 ] Brandon Williams commented on CASSANDRA-3972: - It's not too surprising then that a very long and arduous bisect points toward CASSANDRA-3716. > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3973) Pig CounterColumnFamily support
Pig CounterColumnFamily support --- Key: CASSANDRA-3973 URL: https://issues.apache.org/jira/browse/CASSANDRA-3973 Project: Cassandra Issue Type: Bug Components: Hadoop Affects Versions: 1.0.8 Environment: OSX 10.6.latest, Pig 0.9.2 Reporter: Janne Jalkanen Included a patch to the current test script to demonstrate that CassandraStorage does not support counter columns, as well as a patch to fix the issue. The core reason is that CassandraStorage assumes that counters can be unpacked using CounterColumnType, when in fact the column value is already a Long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3973) Pig CounterColumnFamily support
[ https://issues.apache.org/jira/browse/CASSANDRA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janne Jalkanen updated CASSANDRA-3973: -- Attachment: cassandrastorage.txt testpatch.txt Patch for tests; another patch for CassandraStorage. > Pig CounterColumnFamily support > --- > > Key: CASSANDRA-3973 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3973 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.0.8 > Environment: OSX 10.6.latest, Pig 0.9.2 >Reporter: Janne Jalkanen > Labels: counters, hadoop, pig > Attachments: cassandrastorage.txt, testpatch.txt > > > Included a patch to the current test script to demonstrate that > CassandraStorage does not support counter columns, as well as a patch to fix > the issue. > The core reason is that CassandraStorage assumes that counters can be > unpacked using CounterColumnType, when in fact the column value is already a > Long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3973) Pig CounterColumnFamily support
[ https://issues.apache.org/jira/browse/CASSANDRA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3973: Attachment: 3973-v2.txt v2 to nip counters directly in parseType so we can catch them in super columns too. > Pig CounterColumnFamily support > --- > > Key: CASSANDRA-3973 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3973 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.0.8 > Environment: OSX 10.6.latest, Pig 0.9.2 >Reporter: Janne Jalkanen > Labels: counters, hadoop, pig > Attachments: 3973-v2.txt, cassandrastorage.txt, testpatch.txt > > > Included a patch to the current test script to demonstrate that > CassandraStorage does not support counter columns, as well as a patch to fix > the issue. > The core reason is that CassandraStorage assumes that counters can be > unpacked using CounterColumnType, when in fact the column value is already a > Long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3973) Pig CounterColumnFamily support
[ https://issues.apache.org/jira/browse/CASSANDRA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3973: Reviewer: xedin Assignee: Brandon Williams > Pig CounterColumnFamily support > --- > > Key: CASSANDRA-3973 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3973 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.0.8 > Environment: OSX 10.6.latest, Pig 0.9.2 >Reporter: Janne Jalkanen >Assignee: Brandon Williams > Labels: counters, hadoop, pig > Attachments: 3973-v2.txt, cassandrastorage.txt, testpatch.txt > > > Included a patch to the current test script to demonstrate that > CassandraStorage does not support counter columns, as well as a patch to fix > the issue. > The core reason is that CassandraStorage assumes that counters can be > unpacked using CounterColumnType, when in fact the column value is already a > Long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3973) Pig CounterColumnFamily support
[ https://issues.apache.org/jira/browse/CASSANDRA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janne Jalkanen updated CASSANDRA-3973: -- Attachment: v3.txt V2 patch wasn't working, getDefaultMarshallers() was invoking TypeParser.parse() directly. Here's v3 which fixes that issue (and protects against null reference). > Pig CounterColumnFamily support > --- > > Key: CASSANDRA-3973 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3973 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.0.8 > Environment: OSX 10.6.latest, Pig 0.9.2 >Reporter: Janne Jalkanen >Assignee: Brandon Williams > Labels: counters, hadoop, pig > Attachments: 3973-v2.txt, cassandrastorage.txt, testpatch.txt, v3.txt > > > Included a patch to the current test script to demonstrate that > CassandraStorage does not support counter columns, as well as a patch to fix > the issue. > The core reason is that CassandraStorage assumes that counters can be > unpacked using CounterColumnType, when in fact the column value is already a > Long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3973) Pig CounterColumnFamily support
[ https://issues.apache.org/jira/browse/CASSANDRA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3973: Reviewer: brandon.williams (was: xedin) Assignee: Janne Jalkanen (was: Brandon Williams) > Pig CounterColumnFamily support > --- > > Key: CASSANDRA-3973 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3973 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.0.8 > Environment: OSX 10.6.latest, Pig 0.9.2 >Reporter: Janne Jalkanen >Assignee: Janne Jalkanen > Labels: counters, hadoop, pig > Attachments: 3973-v2.txt, cassandrastorage.txt, testpatch.txt, v3.txt > > > Included a patch to the current test script to demonstrate that > CassandraStorage does not support counter columns, as well as a patch to fix > the issue. > The core reason is that CassandraStorage assumes that counters can be > unpacked using CounterColumnType, when in fact the column value is already a > Long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3974) Per-CF TTL
Per-CF TTL -- Key: CASSANDRA-3974 URL: https://issues.apache.org/jira/browse/CASSANDRA-3974 Project: Cassandra Issue Type: New Feature Reporter: Jonathan Ellis Priority: Minor Per-CF TTL would allow compaction optimizations ("drop an entire sstable's worth of expired data") that we can't do with per-column. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218289#comment-13218289 ] Sylvain Lebresne commented on CASSANDRA-3972: - CASSANDRA-3716 made it so that purge don't use the current time internally and is only based on the column local deletion time and the value of gcBefore. So indeed, ColumnFamilyStore.removeDeleted(c, Integer.MAX_VALUE) will remove every expiring column since it basically means 'remove everything that will be expired at the end of time'. The current time should be used instead as gcBefore. Not sure what is the remaining problem though if it doesn't fix it fully. > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973
Updated Branches: refs/heads/cassandra-1.0 1ed6065b5 -> 4a49b9dcc Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4a49b9dc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4a49b9dc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4a49b9dc Branch: refs/heads/cassandra-1.0 Commit: 4a49b9dcc05d6f06c3354fb37a4a0199a2c283d5 Parents: 1ed6065 Author: Brandon Williams Authored: Tue Feb 28 09:47:26 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:47:26 2012 -0600 -- .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ contrib/pig/test/populate-cli.txt | 18 + contrib/pig/test/test_storage.pig | 10 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java -- diff --git a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java index a2213f7..6d1f76a 100644 --- a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java +++ b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java @@ -209,17 +209,11 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo AbstractType subcomparator; AbstractType default_validator; AbstractType key_validator; -try -{ -comparator = TypeParser.parse(cfDef.getComparator_type()); -subcomparator = TypeParser.parse(cfDef.getSubcomparator_type()); -default_validator = TypeParser.parse(cfDef.getDefault_validation_class()); -key_validator = TypeParser.parse(cfDef.getKey_validation_class()); -} -catch (ConfigurationException e) -{ -throw new IOException(e); -} + +comparator = parseType(cfDef.getComparator_type()); +subcomparator = parseType(cfDef.getSubcomparator_type()); +default_validator = parseType(cfDef.getDefault_validation_class()); +key_validator = parseType(cfDef.getKey_validation_class()); marshallers.add(comparator); marshallers.add(default_validator); @@ -254,6 +248,9 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo { try { +// always treat counters like longs, specifically CCT.compose is not what we need +if (type != null && type.equals("org.apache.cassandra.db.marshal.CounterColumnType")) +return LongType.instance; return TypeParser.parse(type); } catch (ConfigurationException e) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/populate-cli.txt -- diff --git a/contrib/pig/test/populate-cli.txt b/contrib/pig/test/populate-cli.txt index f266ce9..0164afe 100644 --- a/contrib/pig/test/populate-cli.txt +++ b/contrib/pig/test/populate-cli.txt @@ -1,8 +1,6 @@ -create keyspace PigTest; +create keyspace PigTest with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options={replication_factor:1}; use PigTest; create column family SomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -17,8 +15,6 @@ column_metadata = ]; create column family CopyOfSomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -79,4 +75,14 @@ create column family Bytes with comparator = UTF8Type; set U8['foo']['x'] = ascii('Z'); -set Bytes[ascii('foo')]['x'] = ascii('Z') +set Bytes[ascii('foo')]['x'] = ascii('Z'); + +create column family CC with +key_validation_class = UTF8Type and +default_validation_class=CounterColumnType +and comparator=UTF8Type; + +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['fist']; http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/test_storage.pig -- diff --git a/contrib/pig/
[2/2] git commit: Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973
Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4a49b9dc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4a49b9dc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4a49b9dc Branch: refs/heads/cassandra-1.1.0 Commit: 4a49b9dcc05d6f06c3354fb37a4a0199a2c283d5 Parents: 1ed6065 Author: Brandon Williams Authored: Tue Feb 28 09:47:26 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:47:26 2012 -0600 -- .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ contrib/pig/test/populate-cli.txt | 18 + contrib/pig/test/test_storage.pig | 10 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java -- diff --git a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java index a2213f7..6d1f76a 100644 --- a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java +++ b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java @@ -209,17 +209,11 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo AbstractType subcomparator; AbstractType default_validator; AbstractType key_validator; -try -{ -comparator = TypeParser.parse(cfDef.getComparator_type()); -subcomparator = TypeParser.parse(cfDef.getSubcomparator_type()); -default_validator = TypeParser.parse(cfDef.getDefault_validation_class()); -key_validator = TypeParser.parse(cfDef.getKey_validation_class()); -} -catch (ConfigurationException e) -{ -throw new IOException(e); -} + +comparator = parseType(cfDef.getComparator_type()); +subcomparator = parseType(cfDef.getSubcomparator_type()); +default_validator = parseType(cfDef.getDefault_validation_class()); +key_validator = parseType(cfDef.getKey_validation_class()); marshallers.add(comparator); marshallers.add(default_validator); @@ -254,6 +248,9 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo { try { +// always treat counters like longs, specifically CCT.compose is not what we need +if (type != null && type.equals("org.apache.cassandra.db.marshal.CounterColumnType")) +return LongType.instance; return TypeParser.parse(type); } catch (ConfigurationException e) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/populate-cli.txt -- diff --git a/contrib/pig/test/populate-cli.txt b/contrib/pig/test/populate-cli.txt index f266ce9..0164afe 100644 --- a/contrib/pig/test/populate-cli.txt +++ b/contrib/pig/test/populate-cli.txt @@ -1,8 +1,6 @@ -create keyspace PigTest; +create keyspace PigTest with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options={replication_factor:1}; use PigTest; create column family SomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -17,8 +15,6 @@ column_metadata = ]; create column family CopyOfSomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -79,4 +75,14 @@ create column family Bytes with comparator = UTF8Type; set U8['foo']['x'] = ascii('Z'); -set Bytes[ascii('foo')]['x'] = ascii('Z') +set Bytes[ascii('foo')]['x'] = ascii('Z'); + +create column family CC with +key_validation_class = UTF8Type and +default_validation_class=CounterColumnType +and comparator=UTF8Type; + +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['fist']; http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/test_storage.pig -- diff --git a/contrib/pig/test/test_storage.pig b/contrib/pig/test/test_storage.pig index 742
[1/2] git commit: Merge branch 'cassandra-1.0' into cassandra-1.1.0
Updated Branches: refs/heads/cassandra-1.1.0 aa7516847 -> f9454d00b Merge branch 'cassandra-1.0' into cassandra-1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f9454d00 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f9454d00 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f9454d00 Branch: refs/heads/cassandra-1.1.0 Commit: f9454d00b4e4e8520c21fa75b51262a5e46557e1 Parents: aa75168 4a49b9d Author: Brandon Williams Authored: Tue Feb 28 09:49:03 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:49:03 2012 -0600 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9454d00/examples/pig/test/populate-cli.txt -- diff --cc examples/pig/test/populate-cli.txt index f266ce9,000..0164afe mode 100644,00..100644 --- a/examples/pig/test/populate-cli.txt +++ b/examples/pig/test/populate-cli.txt @@@ -1,82 -1,0 +1,88 @@@ - create keyspace PigTest; ++create keyspace PigTest with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options={replication_factor:1}; +use PigTest; +create column family SomeApp with - placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and - strategy_options = {replication_factor:1} and +key_validation_class = UTF8Type and +default_validation_class = LexicalUUIDType and +comparator = UTF8Type and +column_metadata = +[ +{column_name: name, validation_class: UTF8Type, index_type: KEYS}, +{column_name: vote_type, validation_class: UTF8Type}, +{column_name: rating, validation_class: IntegerType}, +{column_name: score, validation_class: LongType}, +{column_name: percent, validation_class: FloatType}, +{column_name: atomic_weight, validation_class: DoubleType}, +]; + +create column family CopyOfSomeApp with - placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and - strategy_options = {replication_factor:1} and +key_validation_class = UTF8Type and +default_validation_class = LexicalUUIDType and +comparator = UTF8Type and +column_metadata = +[ +{column_name: name, validation_class: UTF8Type, index_type: KEYS}, +{column_name: vote_type, validation_class: UTF8Type}, +{column_name: rating, validation_class: IntegerType}, +{column_name: score, validation_class: LongType}, +{column_name: percent, validation_class: FloatType}, +{column_name: atomic_weight, validation_class: DoubleType}, +]; + +set SomeApp['foo']['name'] = 'User Foo'; +set SomeApp['foo']['vote_type'] = 'like'; +set SomeApp['foo']['rating'] = 8; +set SomeApp['foo']['score'] = 125000; +set SomeApp['foo']['percent'] = '85.0'; +set SomeApp['foo']['atomic_weight'] = '2.7182818284590451'; + +set SomeApp['bar']['name'] = 'User Bar'; +set SomeApp['bar']['vote_type'] = 'like'; +set SomeApp['bar']['rating'] = 9; +set SomeApp['bar']['score'] = 15000; +set SomeApp['bar']['percent'] = '35.0'; +set SomeApp['bar']['atomic_weight'] = '3.1415926535897931'; + +set SomeApp['baz']['name'] = 'User Baz'; +set SomeApp['baz']['vote_type'] = 'dislike'; +set SomeApp['baz']['rating'] = 3; +set SomeApp['baz']['score'] = 512000; +set SomeApp['baz']['percent'] = '95.3'; +set SomeApp['baz']['atomic_weight'] = '1.61803399'; +set SomeApp['baz']['extra1'] = lexicaluuid(); +set SomeApp['baz']['extra2'] = lexicaluuid(); +set SomeApp['baz']['extra3'] = lexicaluuid(); + +set SomeApp['qux']['name'] = 'User Qux'; +set SomeApp['qux']['vote_type'] = 'dislike'; +set SomeApp['qux']['rating'] = 2; +set SomeApp['qux']['score'] = 12000; +set SomeApp['qux']['percent'] = '64.7'; +set SomeApp['qux']['atomic_weight'] = '0.660161815846869'; +set SomeApp['qux']['extra1'] = lexicaluuid(); +set SomeApp['qux']['extra2'] = lexicaluuid(); +set SomeApp['qux']['extra3'] = lexicaluuid(); +set SomeApp['qux']['extra4'] = lexicaluuid(); +set SomeApp['qux']['extra5'] = lexicaluuid(); +set SomeApp['qux']['extra6'] = lexicaluuid(); +set SomeApp['qux']['extra7'] = lexicaluuid(); + +create column family U8 with +key_validation_class = UTF8Type and +comparator = UTF8Type; + +create column family Bytes with +key_validation_class = BytesType and +comparator = UTF8Type; + +set U8['foo']['x'] = ascii('Z'); - set Bytes[ascii('foo')]['x'] = ascii('Z') ++set Bytes[ascii('foo')]['x'] = ascii('Z'); ++ ++create column family CC with ++key_validation_class = UTF8Type and ++default_validation_class=CounterColumn
[jira] [Resolved] (CASSANDRA-3973) Pig CounterColumnFamily support
[ https://issues.apache.org/jira/browse/CASSANDRA-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams resolved CASSANDRA-3973. - Resolution: Fixed Fix Version/s: 1.1.0 1.0.9 Committed, thanks! > Pig CounterColumnFamily support > --- > > Key: CASSANDRA-3973 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3973 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.0.8 > Environment: OSX 10.6.latest, Pig 0.9.2 >Reporter: Janne Jalkanen >Assignee: Janne Jalkanen > Labels: counters, hadoop, pig > Fix For: 1.0.9, 1.1.0 > > Attachments: 3973-v2.txt, cassandrastorage.txt, testpatch.txt, v3.txt > > > Included a patch to the current test script to demonstrate that > CassandraStorage does not support counter columns, as well as a patch to fix > the issue. > The core reason is that CassandraStorage assumes that counters can be > unpacked using CounterColumnType, when in fact the column value is already a > Long. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[3/7] git commit: Merge branch 'cassandra-1.1.0' into cassandra-1.1
Merge branch 'cassandra-1.1.0' into cassandra-1.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/86d7c844 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/86d7c844 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/86d7c844 Branch: refs/heads/trunk Commit: 86d7c844c6121e771828846e7aa3089c4f7b3ef6 Parents: d079c82 f9454d0 Author: Brandon Williams Authored: Tue Feb 28 09:50:29 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:50:29 2012 -0600 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/86d7c844/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java --
[4/7] git commit: Merge branch 'cassandra-1.0' into cassandra-1.1.0
Merge branch 'cassandra-1.0' into cassandra-1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f9454d00 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f9454d00 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f9454d00 Branch: refs/heads/cassandra-1.1 Commit: f9454d00b4e4e8520c21fa75b51262a5e46557e1 Parents: aa75168 4a49b9d Author: Brandon Williams Authored: Tue Feb 28 09:49:03 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:49:03 2012 -0600 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9454d00/examples/pig/test/populate-cli.txt -- diff --cc examples/pig/test/populate-cli.txt index f266ce9,000..0164afe mode 100644,00..100644 --- a/examples/pig/test/populate-cli.txt +++ b/examples/pig/test/populate-cli.txt @@@ -1,82 -1,0 +1,88 @@@ - create keyspace PigTest; ++create keyspace PigTest with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options={replication_factor:1}; +use PigTest; +create column family SomeApp with - placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and - strategy_options = {replication_factor:1} and +key_validation_class = UTF8Type and +default_validation_class = LexicalUUIDType and +comparator = UTF8Type and +column_metadata = +[ +{column_name: name, validation_class: UTF8Type, index_type: KEYS}, +{column_name: vote_type, validation_class: UTF8Type}, +{column_name: rating, validation_class: IntegerType}, +{column_name: score, validation_class: LongType}, +{column_name: percent, validation_class: FloatType}, +{column_name: atomic_weight, validation_class: DoubleType}, +]; + +create column family CopyOfSomeApp with - placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and - strategy_options = {replication_factor:1} and +key_validation_class = UTF8Type and +default_validation_class = LexicalUUIDType and +comparator = UTF8Type and +column_metadata = +[ +{column_name: name, validation_class: UTF8Type, index_type: KEYS}, +{column_name: vote_type, validation_class: UTF8Type}, +{column_name: rating, validation_class: IntegerType}, +{column_name: score, validation_class: LongType}, +{column_name: percent, validation_class: FloatType}, +{column_name: atomic_weight, validation_class: DoubleType}, +]; + +set SomeApp['foo']['name'] = 'User Foo'; +set SomeApp['foo']['vote_type'] = 'like'; +set SomeApp['foo']['rating'] = 8; +set SomeApp['foo']['score'] = 125000; +set SomeApp['foo']['percent'] = '85.0'; +set SomeApp['foo']['atomic_weight'] = '2.7182818284590451'; + +set SomeApp['bar']['name'] = 'User Bar'; +set SomeApp['bar']['vote_type'] = 'like'; +set SomeApp['bar']['rating'] = 9; +set SomeApp['bar']['score'] = 15000; +set SomeApp['bar']['percent'] = '35.0'; +set SomeApp['bar']['atomic_weight'] = '3.1415926535897931'; + +set SomeApp['baz']['name'] = 'User Baz'; +set SomeApp['baz']['vote_type'] = 'dislike'; +set SomeApp['baz']['rating'] = 3; +set SomeApp['baz']['score'] = 512000; +set SomeApp['baz']['percent'] = '95.3'; +set SomeApp['baz']['atomic_weight'] = '1.61803399'; +set SomeApp['baz']['extra1'] = lexicaluuid(); +set SomeApp['baz']['extra2'] = lexicaluuid(); +set SomeApp['baz']['extra3'] = lexicaluuid(); + +set SomeApp['qux']['name'] = 'User Qux'; +set SomeApp['qux']['vote_type'] = 'dislike'; +set SomeApp['qux']['rating'] = 2; +set SomeApp['qux']['score'] = 12000; +set SomeApp['qux']['percent'] = '64.7'; +set SomeApp['qux']['atomic_weight'] = '0.660161815846869'; +set SomeApp['qux']['extra1'] = lexicaluuid(); +set SomeApp['qux']['extra2'] = lexicaluuid(); +set SomeApp['qux']['extra3'] = lexicaluuid(); +set SomeApp['qux']['extra4'] = lexicaluuid(); +set SomeApp['qux']['extra5'] = lexicaluuid(); +set SomeApp['qux']['extra6'] = lexicaluuid(); +set SomeApp['qux']['extra7'] = lexicaluuid(); + +create column family U8 with +key_validation_class = UTF8Type and +comparator = UTF8Type; + +create column family Bytes with +key_validation_class = BytesType and +comparator = UTF8Type; + +set U8['foo']['x'] = ascii('Z'); - set Bytes[ascii('foo')]['x'] = ascii('Z') ++set Bytes[ascii('foo')]['x'] = ascii('Z'); ++ ++create column family CC with ++key_validation_class = UTF8Type and ++default_validation_class=CounterColumnType ++and comparator=UTF8Type; ++ ++incr CC['chuck']['kick']; ++incr
[2/7] git commit: Merge branch 'cassandra-1.1.0' into cassandra-1.1
Merge branch 'cassandra-1.1.0' into cassandra-1.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/86d7c844 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/86d7c844 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/86d7c844 Branch: refs/heads/cassandra-1.1 Commit: 86d7c844c6121e771828846e7aa3089c4f7b3ef6 Parents: d079c82 f9454d0 Author: Brandon Williams Authored: Tue Feb 28 09:50:29 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:50:29 2012 -0600 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/86d7c844/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java --
[1/7] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches: refs/heads/cassandra-1.1 d079c8204 -> 86d7c844c refs/heads/trunk d82814ae4 -> cc355ce6b Merge branch 'cassandra-1.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cc355ce6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cc355ce6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cc355ce6 Branch: refs/heads/trunk Commit: cc355ce6b9e14d138d0d1c86660c57c90c8237dd Parents: d82814a 86d7c84 Author: Brandon Williams Authored: Tue Feb 28 09:51:00 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:51:00 2012 -0600 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/cc355ce6/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java --
[5/7] git commit: Merge branch 'cassandra-1.0' into cassandra-1.1.0
Merge branch 'cassandra-1.0' into cassandra-1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f9454d00 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f9454d00 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f9454d00 Branch: refs/heads/trunk Commit: f9454d00b4e4e8520c21fa75b51262a5e46557e1 Parents: aa75168 4a49b9d Author: Brandon Williams Authored: Tue Feb 28 09:49:03 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:49:03 2012 -0600 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9454d00/examples/pig/test/populate-cli.txt -- diff --cc examples/pig/test/populate-cli.txt index f266ce9,000..0164afe mode 100644,00..100644 --- a/examples/pig/test/populate-cli.txt +++ b/examples/pig/test/populate-cli.txt @@@ -1,82 -1,0 +1,88 @@@ - create keyspace PigTest; ++create keyspace PigTest with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options={replication_factor:1}; +use PigTest; +create column family SomeApp with - placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and - strategy_options = {replication_factor:1} and +key_validation_class = UTF8Type and +default_validation_class = LexicalUUIDType and +comparator = UTF8Type and +column_metadata = +[ +{column_name: name, validation_class: UTF8Type, index_type: KEYS}, +{column_name: vote_type, validation_class: UTF8Type}, +{column_name: rating, validation_class: IntegerType}, +{column_name: score, validation_class: LongType}, +{column_name: percent, validation_class: FloatType}, +{column_name: atomic_weight, validation_class: DoubleType}, +]; + +create column family CopyOfSomeApp with - placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and - strategy_options = {replication_factor:1} and +key_validation_class = UTF8Type and +default_validation_class = LexicalUUIDType and +comparator = UTF8Type and +column_metadata = +[ +{column_name: name, validation_class: UTF8Type, index_type: KEYS}, +{column_name: vote_type, validation_class: UTF8Type}, +{column_name: rating, validation_class: IntegerType}, +{column_name: score, validation_class: LongType}, +{column_name: percent, validation_class: FloatType}, +{column_name: atomic_weight, validation_class: DoubleType}, +]; + +set SomeApp['foo']['name'] = 'User Foo'; +set SomeApp['foo']['vote_type'] = 'like'; +set SomeApp['foo']['rating'] = 8; +set SomeApp['foo']['score'] = 125000; +set SomeApp['foo']['percent'] = '85.0'; +set SomeApp['foo']['atomic_weight'] = '2.7182818284590451'; + +set SomeApp['bar']['name'] = 'User Bar'; +set SomeApp['bar']['vote_type'] = 'like'; +set SomeApp['bar']['rating'] = 9; +set SomeApp['bar']['score'] = 15000; +set SomeApp['bar']['percent'] = '35.0'; +set SomeApp['bar']['atomic_weight'] = '3.1415926535897931'; + +set SomeApp['baz']['name'] = 'User Baz'; +set SomeApp['baz']['vote_type'] = 'dislike'; +set SomeApp['baz']['rating'] = 3; +set SomeApp['baz']['score'] = 512000; +set SomeApp['baz']['percent'] = '95.3'; +set SomeApp['baz']['atomic_weight'] = '1.61803399'; +set SomeApp['baz']['extra1'] = lexicaluuid(); +set SomeApp['baz']['extra2'] = lexicaluuid(); +set SomeApp['baz']['extra3'] = lexicaluuid(); + +set SomeApp['qux']['name'] = 'User Qux'; +set SomeApp['qux']['vote_type'] = 'dislike'; +set SomeApp['qux']['rating'] = 2; +set SomeApp['qux']['score'] = 12000; +set SomeApp['qux']['percent'] = '64.7'; +set SomeApp['qux']['atomic_weight'] = '0.660161815846869'; +set SomeApp['qux']['extra1'] = lexicaluuid(); +set SomeApp['qux']['extra2'] = lexicaluuid(); +set SomeApp['qux']['extra3'] = lexicaluuid(); +set SomeApp['qux']['extra4'] = lexicaluuid(); +set SomeApp['qux']['extra5'] = lexicaluuid(); +set SomeApp['qux']['extra6'] = lexicaluuid(); +set SomeApp['qux']['extra7'] = lexicaluuid(); + +create column family U8 with +key_validation_class = UTF8Type and +comparator = UTF8Type; + +create column family Bytes with +key_validation_class = BytesType and +comparator = UTF8Type; + +set U8['foo']['x'] = ascii('Z'); - set Bytes[ascii('foo')]['x'] = ascii('Z') ++set Bytes[ascii('foo')]['x'] = ascii('Z'); ++ ++create column family CC with ++key_validation_class = UTF8Type and ++default_validation_class=CounterColumnType ++and comparator=UTF8Type; ++ ++incr CC['chuck']['kick']; ++incr CC['chuc
[6/7] git commit: Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973
Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4a49b9dc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4a49b9dc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4a49b9dc Branch: refs/heads/trunk Commit: 4a49b9dcc05d6f06c3354fb37a4a0199a2c283d5 Parents: 1ed6065 Author: Brandon Williams Authored: Tue Feb 28 09:47:26 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:47:26 2012 -0600 -- .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ contrib/pig/test/populate-cli.txt | 18 + contrib/pig/test/test_storage.pig | 10 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java -- diff --git a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java index a2213f7..6d1f76a 100644 --- a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java +++ b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java @@ -209,17 +209,11 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo AbstractType subcomparator; AbstractType default_validator; AbstractType key_validator; -try -{ -comparator = TypeParser.parse(cfDef.getComparator_type()); -subcomparator = TypeParser.parse(cfDef.getSubcomparator_type()); -default_validator = TypeParser.parse(cfDef.getDefault_validation_class()); -key_validator = TypeParser.parse(cfDef.getKey_validation_class()); -} -catch (ConfigurationException e) -{ -throw new IOException(e); -} + +comparator = parseType(cfDef.getComparator_type()); +subcomparator = parseType(cfDef.getSubcomparator_type()); +default_validator = parseType(cfDef.getDefault_validation_class()); +key_validator = parseType(cfDef.getKey_validation_class()); marshallers.add(comparator); marshallers.add(default_validator); @@ -254,6 +248,9 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo { try { +// always treat counters like longs, specifically CCT.compose is not what we need +if (type != null && type.equals("org.apache.cassandra.db.marshal.CounterColumnType")) +return LongType.instance; return TypeParser.parse(type); } catch (ConfigurationException e) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/populate-cli.txt -- diff --git a/contrib/pig/test/populate-cli.txt b/contrib/pig/test/populate-cli.txt index f266ce9..0164afe 100644 --- a/contrib/pig/test/populate-cli.txt +++ b/contrib/pig/test/populate-cli.txt @@ -1,8 +1,6 @@ -create keyspace PigTest; +create keyspace PigTest with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options={replication_factor:1}; use PigTest; create column family SomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -17,8 +15,6 @@ column_metadata = ]; create column family CopyOfSomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -79,4 +75,14 @@ create column family Bytes with comparator = UTF8Type; set U8['foo']['x'] = ascii('Z'); -set Bytes[ascii('foo')]['x'] = ascii('Z') +set Bytes[ascii('foo')]['x'] = ascii('Z'); + +create column family CC with +key_validation_class = UTF8Type and +default_validation_class=CounterColumnType +and comparator=UTF8Type; + +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['fist']; http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/test_storage.pig -- diff --git a/contrib/pig/test/test_storage.pig b/contrib/pig/test/test_storage.pig index 7422db4..c49d
[7/7] git commit: Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973
Pig CounterColumnFamily support. Patch by Janne Jalkanen and brandonwilliams, reviewed by brandonwilliams for CASSANDRA-3973 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4a49b9dc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4a49b9dc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4a49b9dc Branch: refs/heads/cassandra-1.1 Commit: 4a49b9dcc05d6f06c3354fb37a4a0199a2c283d5 Parents: 1ed6065 Author: Brandon Williams Authored: Tue Feb 28 09:47:26 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 09:47:26 2012 -0600 -- .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ contrib/pig/test/populate-cli.txt | 18 + contrib/pig/test/test_storage.pig | 10 3 files changed, 30 insertions(+), 17 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java -- diff --git a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java index a2213f7..6d1f76a 100644 --- a/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java +++ b/contrib/pig/src/java/org/apache/cassandra/hadoop/pig/CassandraStorage.java @@ -209,17 +209,11 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo AbstractType subcomparator; AbstractType default_validator; AbstractType key_validator; -try -{ -comparator = TypeParser.parse(cfDef.getComparator_type()); -subcomparator = TypeParser.parse(cfDef.getSubcomparator_type()); -default_validator = TypeParser.parse(cfDef.getDefault_validation_class()); -key_validator = TypeParser.parse(cfDef.getKey_validation_class()); -} -catch (ConfigurationException e) -{ -throw new IOException(e); -} + +comparator = parseType(cfDef.getComparator_type()); +subcomparator = parseType(cfDef.getSubcomparator_type()); +default_validator = parseType(cfDef.getDefault_validation_class()); +key_validator = parseType(cfDef.getKey_validation_class()); marshallers.add(comparator); marshallers.add(default_validator); @@ -254,6 +248,9 @@ public class CassandraStorage extends LoadFunc implements StoreFuncInterface, Lo { try { +// always treat counters like longs, specifically CCT.compose is not what we need +if (type != null && type.equals("org.apache.cassandra.db.marshal.CounterColumnType")) +return LongType.instance; return TypeParser.parse(type); } catch (ConfigurationException e) http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/populate-cli.txt -- diff --git a/contrib/pig/test/populate-cli.txt b/contrib/pig/test/populate-cli.txt index f266ce9..0164afe 100644 --- a/contrib/pig/test/populate-cli.txt +++ b/contrib/pig/test/populate-cli.txt @@ -1,8 +1,6 @@ -create keyspace PigTest; +create keyspace PigTest with placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and strategy_options={replication_factor:1}; use PigTest; create column family SomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -17,8 +15,6 @@ column_metadata = ]; create column family CopyOfSomeApp with -placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and -strategy_options = {replication_factor:1} and key_validation_class = UTF8Type and default_validation_class = LexicalUUIDType and comparator = UTF8Type and @@ -79,4 +75,14 @@ create column family Bytes with comparator = UTF8Type; set U8['foo']['x'] = ascii('Z'); -set Bytes[ascii('foo')]['x'] = ascii('Z') +set Bytes[ascii('foo')]['x'] = ascii('Z'); + +create column family CC with +key_validation_class = UTF8Type and +default_validation_class=CounterColumnType +and comparator=UTF8Type; + +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['kick']; +incr CC['chuck']['fist']; http://git-wip-us.apache.org/repos/asf/cassandra/blob/4a49b9dc/contrib/pig/test/test_storage.pig -- diff --git a/contrib/pig/test/test_storage.pig b/contrib/pig/test/test_storage.pig index 7422d
[jira] [Updated] (CASSANDRA-3442) TTL histogram for sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3442: -- Attachment: 3442-v3.txt v3 attached. It's a bit of two steps forward, one step back: - renames RowStats -> ColumnStats; separates row size computation from column/tombstone counts, moves ColumnStats computation out of serializer and into AbstractCompactedRow - I switched from checking instanceof ExipiringColumn, to instanceof DeletedColumn, since an ExpiringColumn just means it will expire eventually (at which point it turns into a DeletedColumn), whereas a DeletedColumn is a tombstone that will be eligible for dropping after gc_grace_seconds. A common use case for TTL is to expire all data in a row after N days; if we're just going by "this column has a TTL" we'll compact these sstables daily even if none of the data has actually expired yet. Switching to checking for DC instead mitigates this a little. However, the more I think about it, the more I think what we *really* want to track is a histogram of *when tombstones are eligible to be dropped*, relative to the sstable creation time. So, if I had a column that expired after 30 days, and a gc_grace_seconds of 10 days, I'd add an entry for 40 days to the histogram. If I had a new manual delete, I'd add an entry for 10 days. This would allow us to have a good estimate of *how much of the sstable could actually be cleaned out by compaction*, and we could drop the single_compaction_interval code entirely. What do you think? Minor note: the new test seems fairly involved -- what would we lose by just testing compaction of a single sstable w/ tombstones? > TTL histogram for sstable metadata > -- > > Key: CASSANDRA-3442 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3442 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jonathan Ellis >Assignee: Yuki Morishita >Priority: Minor > Labels: compaction > Fix For: 1.2 > > Attachments: 3442-v3.txt, cassandra-1.1-3442.txt > > > Under size-tiered compaction, you can generate large sstables that compact > infrequently. With expiring columns mixed in, we could waste a lot of space > in this situation. > If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a > single-sstable compaction aginst sstables with over 20% (?) expired data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3763) compactionstats throws ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/CASSANDRA-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zenek Kraweznik updated CASSANDRA-3763: --- Priority: Major (was: Minor) Affects Version/s: 1.0.8 > compactionstats throws ArithmeticException: / by zero > - > > Key: CASSANDRA-3763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3763 > Project: Cassandra > Issue Type: Bug > Components: Tools >Affects Versions: 1.0.7, 1.0.8 > Environment: debian linux - openvz kernel, oracle java 1.6.0.26 >Reporter: Zenek Kraweznik > > compactionstats looks like this: > # nodetool -h localhost compactionstats > Exception in thread "main" java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128) > at > org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > # > nodetool is working fine in other actions: > # nodetool -h localhost netstats > Mode: NORMAL > Not sending any streams. > Not receiving any streams. > Pool NameActive Pending Completed > Commandsn/a 0 2 > Responses n/a 0 1810 > # -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3763) compactionstats throws ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/CASSANDRA-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218316#comment-13218316 ] Zenek Kraweznik commented on CASSANDRA-3763: nodetool from all affected version can read compactionstats from cassandra 1.0.6. nodetool from 1.0.6, 1.0.7, 1.0.8 cannot read compacionstats from affected versions of cassandra > compactionstats throws ArithmeticException: / by zero > - > > Key: CASSANDRA-3763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3763 > Project: Cassandra > Issue Type: Bug > Components: Core, Tools >Affects Versions: 1.0.7, 1.0.8 > Environment: debian linux - openvz kernel, oracle java 1.6.0.26 >Reporter: Zenek Kraweznik > > compactionstats looks like this: > # nodetool -h localhost compactionstats > Exception in thread "main" java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128) > at > org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > # > nodetool is working fine in other actions: > # nodetool -h localhost netstats > Mode: NORMAL > Not sending any streams. > Not receiving any streams. > Pool NameActive Pending Completed > Commandsn/a 0 2 > Responses n/a 0 1810 > # -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3763) compactionstats throws ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/CASSANDRA-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zenek Kraweznik updated CASSANDRA-3763: --- Component/s: Core > compactionstats throws ArithmeticException: / by zero > - > > Key: CASSANDRA-3763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3763 > Project: Cassandra > Issue Type: Bug > Components: Core, Tools >Affects Versions: 1.0.7, 1.0.8 > Environment: debian linux - openvz kernel, oracle java 1.6.0.26 >Reporter: Zenek Kraweznik > > compactionstats looks like this: > # nodetool -h localhost compactionstats > Exception in thread "main" java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128) > at > org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > # > nodetool is working fine in other actions: > # nodetool -h localhost netstats > Mode: NORMAL > Not sending any streams. > Not receiving any streams. > Pool NameActive Pending Completed > Commandsn/a 0 2 > Responses n/a 0 1810 > # -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3763) compactionstats throws ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/CASSANDRA-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3763: Priority: Trivial (was: Major) > compactionstats throws ArithmeticException: / by zero > - > > Key: CASSANDRA-3763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3763 > Project: Cassandra > Issue Type: Bug > Components: Core, Tools >Affects Versions: 1.0.7, 1.0.8 > Environment: debian linux - openvz kernel, oracle java 1.6.0.26 >Reporter: Zenek Kraweznik >Priority: Trivial > > compactionstats looks like this: > # nodetool -h localhost compactionstats > Exception in thread "main" java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128) > at > org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > # > nodetool is working fine in other actions: > # nodetool -h localhost netstats > Mode: NORMAL > Not sending any streams. > Not receiving any streams. > Pool NameActive Pending Completed > Commandsn/a 0 2 > Responses n/a 0 1810 > # -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3763) compactionstats throws ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/CASSANDRA-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218322#comment-13218322 ] Jonathan Ellis commented on CASSANDRA-3763: --- I think you're going to need to tell us how to reproduce, because we haven't had any luck yet with the Zen Mind Powers approach. > compactionstats throws ArithmeticException: / by zero > - > > Key: CASSANDRA-3763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3763 > Project: Cassandra > Issue Type: Bug > Components: Core, Tools >Affects Versions: 1.0.7, 1.0.8 > Environment: debian linux - openvz kernel, oracle java 1.6.0.26 >Reporter: Zenek Kraweznik >Priority: Trivial > > compactionstats looks like this: > # nodetool -h localhost compactionstats > Exception in thread "main" java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128) > at > org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > # > nodetool is working fine in other actions: > # nodetool -h localhost netstats > Mode: NORMAL > Not sending any streams. > Not receiving any streams. > Pool NameActive Pending Completed > Commandsn/a 0 2 > Responses n/a 0 1810 > # -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3972: Attachment: 3972.txt That does indeed fix it. Patch to use the current time when filtering tombstones. > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > Attachments: 3972.txt > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-3972: Reviewer: slebresne > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > Attachments: 3972.txt > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218328#comment-13218328 ] Jonathan Ellis commented on CASSANDRA-3972: --- +1 > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > Attachments: 3972.txt > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: Use current time when filtering tombstones in HH. Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-3972
Updated Branches: refs/heads/cassandra-1.1.0 f9454d00b -> 2e7906926 Use current time when filtering tombstones in HH. Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-3972 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e790692 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e790692 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e790692 Branch: refs/heads/cassandra-1.1.0 Commit: 2e79069261fb780dd21c534304abb4ffe1ca422e Parents: f9454d0 Author: Brandon Williams Authored: Tue Feb 28 10:33:28 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 10:33:28 2012 -0600 -- .../apache/cassandra/db/HintedHandOffManager.java |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e790692/src/java/org/apache/cassandra/db/HintedHandOffManager.java -- diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java b/src/java/org/apache/cassandra/db/HintedHandOffManager.java index 8e6acb2..d614a8d 100644 --- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java +++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java @@ -311,7 +311,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean while (true) { QueryFilter filter = QueryFilter.getSliceFilter(epkey, new QueryPath(HINTS_CF), startColumn, ByteBufferUtil.EMPTY_BYTE_BUFFER, false, pageSize); -ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), Integer.MAX_VALUE); +ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), (int)(System.currentTimeMillis() / 1000)); if (pagingFinished(hintsPage, startColumn)) break;
[jira] [Resolved] (CASSANDRA-3972) HintedHandoff fails to deliver any hints
[ https://issues.apache.org/jira/browse/CASSANDRA-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams resolved CASSANDRA-3972. - Resolution: Fixed Reviewer: jbellis (was: slebresne) Committed. > HintedHandoff fails to deliver any hints > > > Key: CASSANDRA-3972 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3972 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.0 >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Blocker > Labels: hintedhandoff > Fix For: 1.1.0 > > Attachments: 3972.txt > > > Summary says it all. Whether in a memtable or sstable, no hints are > delivered. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[1/5] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches: refs/heads/cassandra-1.1 86d7c844c -> f6ba989f9 refs/heads/trunk cc355ce6b -> e0d9ade44 Merge branch 'cassandra-1.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e0d9ade4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e0d9ade4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e0d9ade4 Branch: refs/heads/trunk Commit: e0d9ade447443abaa39f796c1d17b57f6dc79a9e Parents: cc355ce f6ba989 Author: Jonathan Ellis Authored: Tue Feb 28 10:35:14 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:35:14 2012 -0600 -- --
[4/5] git commit: update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971
update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f7629d84 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f7629d84 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f7629d84 Branch: refs/heads/trunk Commit: f7629d8498d4d857128ee1697428d9d5fd9c9273 Parents: f9454d0 Author: Jonathan Ellis Authored: Tue Feb 28 10:34:18 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:34:18 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7629d84/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 4ea8ed1..1022631 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,10 +2668,11 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; + for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[2/5] git commit: merge from 1.1.0
merge from 1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f6ba989f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f6ba989f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f6ba989f Branch: refs/heads/cassandra-1.1 Commit: f6ba989f9a5d65474b82b44450d6a35a9634c924 Parents: 86d7c84 f7629d8 Author: Jonathan Ellis Authored: Tue Feb 28 10:35:00 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:35:00 2012 -0600 -- --
[5/5] git commit: update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971
update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f7629d84 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f7629d84 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f7629d84 Branch: refs/heads/cassandra-1.1 Commit: f7629d8498d4d857128ee1697428d9d5fd9c9273 Parents: f9454d0 Author: Jonathan Ellis Authored: Tue Feb 28 10:34:18 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:34:18 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f7629d84/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 4ea8ed1..1022631 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,10 +2668,11 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; + for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[3/5] git commit: merge from 1.1.0
merge from 1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f6ba989f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f6ba989f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f6ba989f Branch: refs/heads/trunk Commit: f6ba989f9a5d65474b82b44450d6a35a9634c924 Parents: 86d7c84 f7629d8 Author: Jonathan Ellis Authored: Tue Feb 28 10:35:00 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:35:00 2012 -0600 -- --
git commit: update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971
Updated Branches: refs/heads/cassandra-1.1.0 2e7906926 -> 0d40edb40 update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0d40edb4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0d40edb4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0d40edb4 Branch: refs/heads/cassandra-1.1.0 Commit: 0d40edb40f695dbda1e5da6787424ac067a8f081 Parents: 2e79069 Author: Jonathan Ellis Authored: Tue Feb 28 10:34:18 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:36:28 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d40edb4/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 4ea8ed1..1022631 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,10 +2668,11 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; + for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
git commit: Revert "update hasSameReplication to use the List passed to it"
Updated Branches: refs/heads/cassandra-1.1.0 0d40edb40 -> 40617c8be Revert "update hasSameReplication to use the List passed to it" Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/40617c8b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/40617c8b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/40617c8b Branch: refs/heads/cassandra-1.1.0 Commit: 40617c8be6ea85287f1c75d07c87e8315aa1ed4c Parents: 0d40edb Author: Jonathan Ellis Authored: Tue Feb 28 10:37:32 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:37:32 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/40617c8b/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 1022631..4ea8ed1 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,11 +2668,10 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; - for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[jira] [Created] (CASSANDRA-3975) Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily
Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily -- Key: CASSANDRA-3975 URL: https://issues.apache.org/jira/browse/CASSANDRA-3975 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.7 Reporter: Chris Herron Fix For: 1.0.9, 1.1.0 If hints have accumulated for a CF that has been deleted, Hinted Handoff repeatedly fails until manual intervention removes those hints. For 1.0.7, UnserializableColumnFamilyException is thrown only when a CFid is unknown on the sending node. As discussed on #cassandra-dev, if the schema is in agreement, the affected hint(s) should be deleted to avoid indefinite repeat failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3975) Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily
[ https://issues.apache.org/jira/browse/CASSANDRA-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218348#comment-13218348 ] Chris Herron commented on CASSANDRA-3975: - Stack trace from 1.0.7: INFO [HintedHandoff:854] 2012-02-27 22:39:51,183 org.apache.cassandra.db.HintedHandOffManager Started hinted handoff for token: Token(bytes[7f]) with IP: /XX.XX.XX.XXX ERROR [HintedHandoff:854] 2012-02-27 22:39:51,186 org.apache.cassandra.service.AbstractCassandraDaemon Fatal exception in thread Thread[HintedHandoff:854,1,main] java.lang.RuntimeException: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=2391 at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=2391 at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:401) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:409) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:344) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:248) at org.apache.cassandra.db.HintedHandOffManager.access$200(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:416) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily > -- > > Key: CASSANDRA-3975 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3975 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.7 >Reporter: Chris Herron > Fix For: 1.0.9, 1.1.0 > > > If hints have accumulated for a CF that has been deleted, Hinted Handoff > repeatedly fails until manual intervention removes those hints. For 1.0.7, > UnserializableColumnFamilyException is thrown only when a CFid is unknown on > the sending node. As discussed on #cassandra-dev, if the schema is in > agreement, the affected hint(s) should be deleted to avoid indefinite repeat > failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3975) Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily
[ https://issues.apache.org/jira/browse/CASSANDRA-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218353#comment-13218353 ] Chris Herron commented on CASSANDRA-3975: - Suggest that for 1.0.x, UnserializableColumnFamilyException might be better named UnknownColumnFamilyException (since unknown cfId is the only time its ever thrown. It could then be caught within HHOM.deliverHintsToEndpointInternal and the affected hint deleted. Its unclear to me if this needs to be tackled differently for 1.1. > Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily > -- > > Key: CASSANDRA-3975 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3975 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.7 >Reporter: Chris Herron > Fix For: 1.0.9, 1.1.0 > > > If hints have accumulated for a CF that has been deleted, Hinted Handoff > repeatedly fails until manual intervention removes those hints. For 1.0.7, > UnserializableColumnFamilyException is thrown only when a CFid is unknown on > the sending node. As discussed on #cassandra-dev, if the schema is in > agreement, the affected hint(s) should be deleted to avoid indefinite repeat > failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3965) Cassandra code base has files with import statements having "*" causing compilation failure
[ https://issues.apache.org/jira/browse/CASSANDRA-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218390#comment-13218390 ] Harish Doddi commented on CASSANDRA-3965: - Yes seems like it is fixed in trunk. Thanks I verified it. I could not find any ticket filed in the jira for this issue? > Cassandra code base has files with import statements having "*" causing > compilation failure > --- > > Key: CASSANDRA-3965 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3965 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.0.8 > Environment: Linux >Reporter: Harish Doddi > > I tried to download a jar as part of the new unit test I am writing. I ran my > unit test successfully but later if I run "ant" without ant clean, I run into > the following compilation issue > {code} > build-project: > [echo] apache-cassandra: /Users/harish/workspace/cassandra/build.xml > [javac] Compiling 40 source files to > /Users/harish/workspace/cassandra/build/classes/thrift > [javac] Note: Some input files use unchecked or unsafe operations. > [javac] Note: Recompile with -Xlint:unchecked for details. > [javac] Compiling 568 source files to > /Users/harish/workspace/cassandra/build/classes/main > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/db/ColumnFamilyStore.java:1607: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Iterable> keySamples(Range > range) > [javac] ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:196: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Collection> getLocalRanges(String table) > [javac] ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:201: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Range getLocalPrimaryRange() > [javac]^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:912: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Map, List> > getRangeToAddressMap(String keyspace) > [javac]^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1009: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Map, List> > constructRangeToEndpointMap(String keyspace, List> ranges) > [javac] > ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1009: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Map, List> > constructRangeToEndpointMap(String keyspace, List> ranges) > [javac] ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1438: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Multimap> > getNewSourceRanges(String table, Set> ranges) > [javac] >^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1438: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Multimap> > getNewSourceRanges(String table, Set> ranges) > [javac]
[jira] [Commented] (CASSANDRA-3763) compactionstats throws ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/CASSANDRA-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218395#comment-13218395 ] Zenek Kraweznik commented on CASSANDRA-3763: Problem appears after ugprading 1.0.6 to 1.0.7. I've create only one keyspace with 4 column family. All of my CF are: with column_type = 'Standard' and comparator = 'AsciiType' and default_validation_class = 'AsciiType' and key_validation_class = 'AsciiType' and rows_cached = 0.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 20.0 and key_cache_save_period = 14400 and read_repair_chance = 1.0 and gc_grace = 864000 and min_compaction_threshold = 2 and max_compaction_threshold = 4 and replicate_on_write = true and row_cache_provider = 'ConcurrentLinkedHashCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy' and compaction_strategy_options = {'sstable_size_in_mb' : '4096'} and compression_options = {'chunk_length_kb' : '64', 'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'}; system keyspace is untouched, I didn't change it. After upgrading node try to use 'nodetool -h $IP compactionstats' > compactionstats throws ArithmeticException: / by zero > - > > Key: CASSANDRA-3763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3763 > Project: Cassandra > Issue Type: Bug > Components: Core, Tools >Affects Versions: 1.0.7, 1.0.8 > Environment: debian linux - openvz kernel, oracle java 1.6.0.26 >Reporter: Zenek Kraweznik >Priority: Trivial > > compactionstats looks like this: > # nodetool -h localhost compactionstats > Exception in thread "main" java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128) > at > org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662)
[2/2] git commit: Fix race between writes and read for cache
Fix race between writes and read for cache patch by jbellis and slebresne; reviewed by jbellis and slebresne for CASSANDRA-3862 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c9270f4e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c9270f4e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c9270f4e Branch: refs/heads/cassandra-1.1.0 Commit: c9270f4e3ae5f94d46070f1c7e585c90bc68df7c Parents: aa75168 Author: Sylvain Lebresne Authored: Tue Feb 28 18:53:32 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:53:32 2012 +0100 -- CHANGES.txt|1 + .../cassandra/cache/ConcurrentLinkedHashCache.java | 10 ++ .../cache/ConcurrentLinkedHashCacheProvider.java | 13 +- src/java/org/apache/cassandra/cache/ICache.java|4 + .../org/apache/cassandra/cache/IRowCacheEntry.java |5 + .../apache/cassandra/cache/IRowCacheProvider.java |2 +- .../apache/cassandra/cache/InstrumentingCache.java | 10 ++ .../apache/cassandra/cache/RowCacheSentinel.java | 45 ++ .../apache/cassandra/cache/SerializingCache.java | 35 - .../cassandra/cache/SerializingCacheProvider.java | 47 ++- src/java/org/apache/cassandra/db/ColumnFamily.java |9 +- .../org/apache/cassandra/db/ColumnFamilyStore.java | 119 +++ .../apache/cassandra/db/RowIteratorFactory.java|2 + .../db/compaction/CompactionIterable.java |2 +- .../db/compaction/ParallelCompactionIterable.java |2 +- .../org/apache/cassandra/service/CacheService.java |8 +- .../cassandra/streaming/IncomingStreamReader.java |3 +- .../org/apache/cassandra/utils/StatusLogger.java |3 +- .../unit/org/apache/cassandra/db/RowCacheTest.java |4 +- 19 files changed, 264 insertions(+), 60 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 41316eb..b5b79a9 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -14,6 +14,7 @@ * ignore deprecated KsDef/CfDef/ColumnDef fields in native schema (CASSANDRA-3963) * CLI to report when unsupported column_metadata pair was given (CASSANDRA-3959) * reincarnate removed and deprecated KsDef/CfDef attributes (CASSANDRA-3953) + * Fix race between writes and read for cache (CASSANDRA-3862) Merged from 1.0: * remove the wait on hint future during write (CASSANDRA-3870) * (cqlsh) ignore missing CfDef opts (CASSANDRA-3933) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java -- diff --git a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java index 8f4d2f0..a1cf4ea 100644 --- a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java +++ b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java @@ -117,6 +117,16 @@ public class ConcurrentLinkedHashCache implements ICache map.put(key, value); } +public boolean putIfAbsent(K key, V value) +{ +return map.putIfAbsent(key, value) == null; +} + +public boolean replace(K key, V old, V value) +{ +return map.replace(key, old, value); +} + public void remove(K key) { map.remove(key); http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java -- diff --git a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java index 851d4c5..71babd6 100644 --- a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java +++ b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java @@ -20,8 +20,6 @@ package org.apache.cassandra.cache; * */ -import org.apache.cassandra.db.ColumnFamily; - import com.googlecode.concurrentlinkedhashmap.Weigher; import com.googlecode.concurrentlinkedhashmap.Weighers; @@ -29,21 +27,20 @@ import org.github.jamm.MemoryMeter; public class ConcurrentLinkedHashCacheProvider implements IRowCacheProvider { -public ICache create(int capacity, boolean useMemoryWeigher) +public ICache create(int capacity, boolean useMemoryWeigher) { return ConcurrentLinkedHashCache.create(capacity, useMemoryWeigher ? createMemoryWeigher() -
[1/2] git commit: Merge branch 'cassandra-1.1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.1.0
Updated Branches: refs/heads/cassandra-1.1.0 40617c8be -> 4fcb56fa3 Merge branch 'cassandra-1.1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4fcb56fa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4fcb56fa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4fcb56fa Branch: refs/heads/cassandra-1.1.0 Commit: 4fcb56fa3a4a59937c21dc4b17ecee9e311425d2 Parents: c9270f4 40617c8 Author: Sylvain Lebresne Authored: Tue Feb 28 18:55:01 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:55:01 2012 +0100 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../apache/cassandra/db/HintedHandOffManager.java |2 +- .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 4 files changed, 31 insertions(+), 18 deletions(-) --
[4/6] git commit: Revert "update hasSameReplication to use the List passed to it"
Revert "update hasSameReplication to use the List passed to it" Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/40617c8b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/40617c8b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/40617c8b Branch: refs/heads/cassandra-1.1 Commit: 40617c8be6ea85287f1c75d07c87e8315aa1ed4c Parents: 0d40edb Author: Jonathan Ellis Authored: Tue Feb 28 10:37:32 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:37:32 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/40617c8b/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 1022631..4ea8ed1 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,11 +2668,10 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; - for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[1/6] git commit: Merge branch 'cassandra-1.1.0' into cassandra-1.1
Updated Branches: refs/heads/cassandra-1.1 f6ba989f9 -> 2cc612ca0 Merge branch 'cassandra-1.1.0' into cassandra-1.1 Conflicts: src/java/org/apache/cassandra/db/ColumnFamilyStore.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2cc612ca Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2cc612ca Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2cc612ca Branch: refs/heads/cassandra-1.1 Commit: 2cc612ca037fedb0a3d03660ac4ba76be9105703 Parents: f6ba989 4fcb56f Author: Sylvain Lebresne Authored: Tue Feb 28 18:58:37 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:58:37 2012 +0100 -- CHANGES.txt|1 + .../cassandra/cache/ConcurrentLinkedHashCache.java | 10 ++ .../cache/ConcurrentLinkedHashCacheProvider.java | 13 +- src/java/org/apache/cassandra/cache/ICache.java|4 + .../org/apache/cassandra/cache/IRowCacheEntry.java |5 + .../apache/cassandra/cache/IRowCacheProvider.java |2 +- .../apache/cassandra/cache/InstrumentingCache.java | 10 ++ .../apache/cassandra/cache/RowCacheSentinel.java | 45 ++ .../apache/cassandra/cache/SerializingCache.java | 35 - .../cassandra/cache/SerializingCacheProvider.java | 47 ++- src/java/org/apache/cassandra/db/ColumnFamily.java |9 +- .../org/apache/cassandra/db/ColumnFamilyStore.java | 120 +++ .../apache/cassandra/db/HintedHandOffManager.java |2 +- .../apache/cassandra/db/RowIteratorFactory.java|2 + .../db/compaction/CompactionIterable.java |2 +- .../db/compaction/ParallelCompactionIterable.java |2 +- .../org/apache/cassandra/service/CacheService.java |8 +- .../cassandra/streaming/IncomingStreamReader.java |3 +- .../org/apache/cassandra/utils/StatusLogger.java |3 +- .../unit/org/apache/cassandra/db/RowCacheTest.java |4 +- 20 files changed, 265 insertions(+), 62 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/db/RowIteratorFactory.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/db/compaction/ParallelCompactionIterable.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/service/CacheService.java --
[3/6] git commit: Fix race between writes and read for cache
Fix race between writes and read for cache patch by jbellis and slebresne; reviewed by jbellis and slebresne for CASSANDRA-3862 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c9270f4e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c9270f4e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c9270f4e Branch: refs/heads/cassandra-1.1 Commit: c9270f4e3ae5f94d46070f1c7e585c90bc68df7c Parents: aa75168 Author: Sylvain Lebresne Authored: Tue Feb 28 18:53:32 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:53:32 2012 +0100 -- CHANGES.txt|1 + .../cassandra/cache/ConcurrentLinkedHashCache.java | 10 ++ .../cache/ConcurrentLinkedHashCacheProvider.java | 13 +- src/java/org/apache/cassandra/cache/ICache.java|4 + .../org/apache/cassandra/cache/IRowCacheEntry.java |5 + .../apache/cassandra/cache/IRowCacheProvider.java |2 +- .../apache/cassandra/cache/InstrumentingCache.java | 10 ++ .../apache/cassandra/cache/RowCacheSentinel.java | 45 ++ .../apache/cassandra/cache/SerializingCache.java | 35 - .../cassandra/cache/SerializingCacheProvider.java | 47 ++- src/java/org/apache/cassandra/db/ColumnFamily.java |9 +- .../org/apache/cassandra/db/ColumnFamilyStore.java | 119 +++ .../apache/cassandra/db/RowIteratorFactory.java|2 + .../db/compaction/CompactionIterable.java |2 +- .../db/compaction/ParallelCompactionIterable.java |2 +- .../org/apache/cassandra/service/CacheService.java |8 +- .../cassandra/streaming/IncomingStreamReader.java |3 +- .../org/apache/cassandra/utils/StatusLogger.java |3 +- .../unit/org/apache/cassandra/db/RowCacheTest.java |4 +- 19 files changed, 264 insertions(+), 60 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 41316eb..b5b79a9 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -14,6 +14,7 @@ * ignore deprecated KsDef/CfDef/ColumnDef fields in native schema (CASSANDRA-3963) * CLI to report when unsupported column_metadata pair was given (CASSANDRA-3959) * reincarnate removed and deprecated KsDef/CfDef attributes (CASSANDRA-3953) + * Fix race between writes and read for cache (CASSANDRA-3862) Merged from 1.0: * remove the wait on hint future during write (CASSANDRA-3870) * (cqlsh) ignore missing CfDef opts (CASSANDRA-3933) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java -- diff --git a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java index 8f4d2f0..a1cf4ea 100644 --- a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java +++ b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java @@ -117,6 +117,16 @@ public class ConcurrentLinkedHashCache implements ICache map.put(key, value); } +public boolean putIfAbsent(K key, V value) +{ +return map.putIfAbsent(key, value) == null; +} + +public boolean replace(K key, V old, V value) +{ +return map.replace(key, old, value); +} + public void remove(K key) { map.remove(key); http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java -- diff --git a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java index 851d4c5..71babd6 100644 --- a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java +++ b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java @@ -20,8 +20,6 @@ package org.apache.cassandra.cache; * */ -import org.apache.cassandra.db.ColumnFamily; - import com.googlecode.concurrentlinkedhashmap.Weigher; import com.googlecode.concurrentlinkedhashmap.Weighers; @@ -29,21 +27,20 @@ import org.github.jamm.MemoryMeter; public class ConcurrentLinkedHashCacheProvider implements IRowCacheProvider { -public ICache create(int capacity, boolean useMemoryWeigher) +public ICache create(int capacity, boolean useMemoryWeigher) { return ConcurrentLinkedHashCache.create(capacity, useMemoryWeigher ? createMemoryWeigher() -
[2/6] git commit: Merge branch 'cassandra-1.1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.1.0
Merge branch 'cassandra-1.1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4fcb56fa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4fcb56fa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4fcb56fa Branch: refs/heads/cassandra-1.1 Commit: 4fcb56fa3a4a59937c21dc4b17ecee9e311425d2 Parents: c9270f4 40617c8 Author: Sylvain Lebresne Authored: Tue Feb 28 18:55:01 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:55:01 2012 +0100 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../apache/cassandra/db/HintedHandOffManager.java |2 +- .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 4 files changed, 31 insertions(+), 18 deletions(-) --
[5/6] git commit: update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971
update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0d40edb4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0d40edb4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0d40edb4 Branch: refs/heads/cassandra-1.1 Commit: 0d40edb40f695dbda1e5da6787424ac067a8f081 Parents: 2e79069 Author: Jonathan Ellis Authored: Tue Feb 28 10:34:18 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:36:28 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d40edb4/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 4ea8ed1..1022631 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,10 +2668,11 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; + for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[6/6] git commit: Use current time when filtering tombstones in HH. Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-3972
Use current time when filtering tombstones in HH. Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-3972 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e790692 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e790692 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e790692 Branch: refs/heads/cassandra-1.1 Commit: 2e79069261fb780dd21c534304abb4ffe1ca422e Parents: f9454d0 Author: Brandon Williams Authored: Tue Feb 28 10:33:28 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 10:33:28 2012 -0600 -- .../apache/cassandra/db/HintedHandOffManager.java |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e790692/src/java/org/apache/cassandra/db/HintedHandOffManager.java -- diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java b/src/java/org/apache/cassandra/db/HintedHandOffManager.java index 8e6acb2..d614a8d 100644 --- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java +++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java @@ -311,7 +311,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean while (true) { QueryFilter filter = QueryFilter.getSliceFilter(epkey, new QueryPath(HINTS_CF), startColumn, ByteBufferUtil.EMPTY_BYTE_BUFFER, false, pageSize); -ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), Integer.MAX_VALUE); +ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), (int)(System.currentTimeMillis() / 1000)); if (pagingFinished(hintsPage, startColumn)) break;
[5/7] git commit: Revert "update hasSameReplication to use the List passed to it"
Revert "update hasSameReplication to use the List passed to it" Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/40617c8b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/40617c8b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/40617c8b Branch: refs/heads/trunk Commit: 40617c8be6ea85287f1c75d07c87e8315aa1ed4c Parents: 0d40edb Author: Jonathan Ellis Authored: Tue Feb 28 10:37:32 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:37:32 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/40617c8b/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 1022631..4ea8ed1 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,11 +2668,10 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; - for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[2/7] git commit: Merge branch 'cassandra-1.1.0' into cassandra-1.1
Merge branch 'cassandra-1.1.0' into cassandra-1.1 Conflicts: src/java/org/apache/cassandra/db/ColumnFamilyStore.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2cc612ca Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2cc612ca Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2cc612ca Branch: refs/heads/trunk Commit: 2cc612ca037fedb0a3d03660ac4ba76be9105703 Parents: f6ba989 4fcb56f Author: Sylvain Lebresne Authored: Tue Feb 28 18:58:37 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:58:37 2012 +0100 -- CHANGES.txt|1 + .../cassandra/cache/ConcurrentLinkedHashCache.java | 10 ++ .../cache/ConcurrentLinkedHashCacheProvider.java | 13 +- src/java/org/apache/cassandra/cache/ICache.java|4 + .../org/apache/cassandra/cache/IRowCacheEntry.java |5 + .../apache/cassandra/cache/IRowCacheProvider.java |2 +- .../apache/cassandra/cache/InstrumentingCache.java | 10 ++ .../apache/cassandra/cache/RowCacheSentinel.java | 45 ++ .../apache/cassandra/cache/SerializingCache.java | 35 - .../cassandra/cache/SerializingCacheProvider.java | 47 ++- src/java/org/apache/cassandra/db/ColumnFamily.java |9 +- .../org/apache/cassandra/db/ColumnFamilyStore.java | 120 +++ .../apache/cassandra/db/HintedHandOffManager.java |2 +- .../apache/cassandra/db/RowIteratorFactory.java|2 + .../db/compaction/CompactionIterable.java |2 +- .../db/compaction/ParallelCompactionIterable.java |2 +- .../org/apache/cassandra/service/CacheService.java |8 +- .../cassandra/streaming/IncomingStreamReader.java |3 +- .../org/apache/cassandra/utils/StatusLogger.java |3 +- .../unit/org/apache/cassandra/db/RowCacheTest.java |4 +- 20 files changed, 265 insertions(+), 62 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/db/ColumnFamilyStore.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/db/RowIteratorFactory.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/db/compaction/ParallelCompactionIterable.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2cc612ca/src/java/org/apache/cassandra/service/CacheService.java --
[1/7] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches: refs/heads/trunk e0d9ade44 -> 5c7ccb033 Merge branch 'cassandra-1.1' into trunk Conflicts: src/java/org/apache/cassandra/db/ColumnFamily.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5c7ccb03 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5c7ccb03 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5c7ccb03 Branch: refs/heads/trunk Commit: 5c7ccb033827f4d041f9e6ff2e4641163362f819 Parents: e0d9ade 2cc612c Author: Sylvain Lebresne Authored: Tue Feb 28 19:02:49 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 19:02:49 2012 +0100 -- CHANGES.txt|1 + .../cassandra/cache/ConcurrentLinkedHashCache.java | 10 ++ .../cache/ConcurrentLinkedHashCacheProvider.java | 13 +- src/java/org/apache/cassandra/cache/ICache.java|4 + .../org/apache/cassandra/cache/IRowCacheEntry.java |5 + .../apache/cassandra/cache/IRowCacheProvider.java |2 +- .../apache/cassandra/cache/InstrumentingCache.java | 10 ++ .../apache/cassandra/cache/RowCacheSentinel.java | 45 ++ .../apache/cassandra/cache/SerializingCache.java | 35 - .../cassandra/cache/SerializingCacheProvider.java | 47 ++- src/java/org/apache/cassandra/db/ColumnFamily.java |7 +- .../org/apache/cassandra/db/ColumnFamilyStore.java | 120 +++ .../apache/cassandra/db/HintedHandOffManager.java |2 +- .../apache/cassandra/db/RowIteratorFactory.java|2 + .../db/compaction/CompactionIterable.java |2 +- .../db/compaction/ParallelCompactionIterable.java |2 +- .../org/apache/cassandra/service/CacheService.java |8 +- .../cassandra/streaming/IncomingStreamReader.java |3 +- .../org/apache/cassandra/utils/StatusLogger.java |3 +- .../unit/org/apache/cassandra/db/RowCacheTest.java |4 +- 20 files changed, 264 insertions(+), 61 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c7ccb03/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c7ccb03/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java -- diff --cc src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java index 0c00539,71babd6..6367364 --- a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java +++ b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java @@@ -7,18 -9,17 +7,16 @@@ * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. + * http://www.apache.org/licenses/LICENSE-2.0 * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. */ +package org.apache.cassandra.cache; - import org.apache.cassandra.db.ColumnFamily; - import com.googlecode.concurrentlinkedhashmap.Weigher; import com.googlecode.concurrentlinkedhashmap.Weighers; http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c7ccb03/src/java/org/apache/cassandra/cache/ICache.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c7ccb03/src/java/org/apache/cassandra/cache/IRowCacheProvider.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c7ccb03/src/java/org/apache/cassandra/cache/InstrumentingCache.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c7ccb03/src/java/org/apache/cassandra/cache/SerializingCache.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/5c7ccb03/src/java/org/apache/cassandra/cache/SerializingCacheProvider.java -- diff --cc src/java/org/apache/cassandra/ca
[3/7] git commit: Merge branch 'cassandra-1.1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.1.0
Merge branch 'cassandra-1.1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4fcb56fa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4fcb56fa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4fcb56fa Branch: refs/heads/trunk Commit: 4fcb56fa3a4a59937c21dc4b17ecee9e311425d2 Parents: c9270f4 40617c8 Author: Sylvain Lebresne Authored: Tue Feb 28 18:55:01 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:55:01 2012 +0100 -- examples/pig/test/populate-cli.txt | 18 + examples/pig/test/test_storage.pig | 10 .../apache/cassandra/db/HintedHandOffManager.java |2 +- .../cassandra/hadoop/pig/CassandraStorage.java | 19 ++ 4 files changed, 31 insertions(+), 18 deletions(-) --
[4/7] git commit: Fix race between writes and read for cache
Fix race between writes and read for cache patch by jbellis and slebresne; reviewed by jbellis and slebresne for CASSANDRA-3862 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c9270f4e Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c9270f4e Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c9270f4e Branch: refs/heads/trunk Commit: c9270f4e3ae5f94d46070f1c7e585c90bc68df7c Parents: aa75168 Author: Sylvain Lebresne Authored: Tue Feb 28 18:53:32 2012 +0100 Committer: Sylvain Lebresne Committed: Tue Feb 28 18:53:32 2012 +0100 -- CHANGES.txt|1 + .../cassandra/cache/ConcurrentLinkedHashCache.java | 10 ++ .../cache/ConcurrentLinkedHashCacheProvider.java | 13 +- src/java/org/apache/cassandra/cache/ICache.java|4 + .../org/apache/cassandra/cache/IRowCacheEntry.java |5 + .../apache/cassandra/cache/IRowCacheProvider.java |2 +- .../apache/cassandra/cache/InstrumentingCache.java | 10 ++ .../apache/cassandra/cache/RowCacheSentinel.java | 45 ++ .../apache/cassandra/cache/SerializingCache.java | 35 - .../cassandra/cache/SerializingCacheProvider.java | 47 ++- src/java/org/apache/cassandra/db/ColumnFamily.java |9 +- .../org/apache/cassandra/db/ColumnFamilyStore.java | 119 +++ .../apache/cassandra/db/RowIteratorFactory.java|2 + .../db/compaction/CompactionIterable.java |2 +- .../db/compaction/ParallelCompactionIterable.java |2 +- .../org/apache/cassandra/service/CacheService.java |8 +- .../cassandra/streaming/IncomingStreamReader.java |3 +- .../org/apache/cassandra/utils/StatusLogger.java |3 +- .../unit/org/apache/cassandra/db/RowCacheTest.java |4 +- 19 files changed, 264 insertions(+), 60 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 41316eb..b5b79a9 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -14,6 +14,7 @@ * ignore deprecated KsDef/CfDef/ColumnDef fields in native schema (CASSANDRA-3963) * CLI to report when unsupported column_metadata pair was given (CASSANDRA-3959) * reincarnate removed and deprecated KsDef/CfDef attributes (CASSANDRA-3953) + * Fix race between writes and read for cache (CASSANDRA-3862) Merged from 1.0: * remove the wait on hint future during write (CASSANDRA-3870) * (cqlsh) ignore missing CfDef opts (CASSANDRA-3933) http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java -- diff --git a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java index 8f4d2f0..a1cf4ea 100644 --- a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java +++ b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCache.java @@ -117,6 +117,16 @@ public class ConcurrentLinkedHashCache implements ICache map.put(key, value); } +public boolean putIfAbsent(K key, V value) +{ +return map.putIfAbsent(key, value) == null; +} + +public boolean replace(K key, V old, V value) +{ +return map.replace(key, old, value); +} + public void remove(K key) { map.remove(key); http://git-wip-us.apache.org/repos/asf/cassandra/blob/c9270f4e/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java -- diff --git a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java index 851d4c5..71babd6 100644 --- a/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java +++ b/src/java/org/apache/cassandra/cache/ConcurrentLinkedHashCacheProvider.java @@ -20,8 +20,6 @@ package org.apache.cassandra.cache; * */ -import org.apache.cassandra.db.ColumnFamily; - import com.googlecode.concurrentlinkedhashmap.Weigher; import com.googlecode.concurrentlinkedhashmap.Weighers; @@ -29,21 +27,20 @@ import org.github.jamm.MemoryMeter; public class ConcurrentLinkedHashCacheProvider implements IRowCacheProvider { -public ICache create(int capacity, boolean useMemoryWeigher) +public ICache create(int capacity, boolean useMemoryWeigher) { return ConcurrentLinkedHashCache.create(capacity, useMemoryWeigher ? createMemoryWeigher() -: Wei
[7/7] git commit: Use current time when filtering tombstones in HH. Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-3972
Use current time when filtering tombstones in HH. Patch by brandonwilliams, reviewed by jbellis for CASSANDRA-3972 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2e790692 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2e790692 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2e790692 Branch: refs/heads/trunk Commit: 2e79069261fb780dd21c534304abb4ffe1ca422e Parents: f9454d0 Author: Brandon Williams Authored: Tue Feb 28 10:33:28 2012 -0600 Committer: Brandon Williams Committed: Tue Feb 28 10:33:28 2012 -0600 -- .../apache/cassandra/db/HintedHandOffManager.java |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2e790692/src/java/org/apache/cassandra/db/HintedHandOffManager.java -- diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java b/src/java/org/apache/cassandra/db/HintedHandOffManager.java index 8e6acb2..d614a8d 100644 --- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java +++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java @@ -311,7 +311,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean while (true) { QueryFilter filter = QueryFilter.getSliceFilter(epkey, new QueryPath(HINTS_CF), startColumn, ByteBufferUtil.EMPTY_BYTE_BUFFER, false, pageSize); -ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), Integer.MAX_VALUE); +ColumnFamily hintsPage = ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), (int)(System.currentTimeMillis() / 1000)); if (pagingFinished(hintsPage, startColumn)) break;
[6/7] git commit: update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971
update hasSameReplication to use the List passed to it patch by dbrosius; reviewed by jbellis for CASSANDRA-3971 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0d40edb4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0d40edb4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0d40edb4 Branch: refs/heads/trunk Commit: 0d40edb40f695dbda1e5da6787424ac067a8f081 Parents: 2e79069 Author: Jonathan Ellis Authored: Tue Feb 28 10:34:18 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 10:36:28 2012 -0600 -- .../apache/cassandra/service/StorageService.java |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0d40edb4/src/java/org/apache/cassandra/service/StorageService.java -- diff --git a/src/java/org/apache/cassandra/service/StorageService.java b/src/java/org/apache/cassandra/service/StorageService.java index 4ea8ed1..1022631 100644 --- a/src/java/org/apache/cassandra/service/StorageService.java +++ b/src/java/org/apache/cassandra/service/StorageService.java @@ -2668,10 +2668,11 @@ public class StorageService implements IEndpointStateChangeSubscriber, StorageSe { if (list.isEmpty()) return false; + for (int i = 0; i < list.size() -1; i++) { -KSMetaData ksm1 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i)); -KSMetaData ksm2 = Schema.instance.getKSMetaData(Schema.instance.getNonSystemTables().get(i + 1)); +KSMetaData ksm1 = Schema.instance.getKSMetaData(list.get(i)); +KSMetaData ksm2 = Schema.instance.getKSMetaData(list.get(i + 1)); if (!ksm1.strategyClass.equals(ksm2.strategyClass) || !Iterators.elementsEqual(ksm1.strategyOptions.entrySet().iterator(), ksm2.strategyOptions.entrySet().iterator()))
[jira] [Resolved] (CASSANDRA-3862) RowCache misses Updates
[ https://issues.apache.org/jira/browse/CASSANDRA-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne resolved CASSANDRA-3862. - Resolution: Fixed Committed, thanks > RowCache misses Updates > --- > > Key: CASSANDRA-3862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3862 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.6 >Reporter: Daniel Doubleday >Assignee: Sylvain Lebresne > Fix For: 1.1.0 > > Attachments: 3862-7.txt, 3862-cleanup.txt, 3862-v2.patch, > 3862-v4.patch, 3862-v5.txt, 3862-v6.txt, 3862-v8.txt, 3862.patch, > 3862_v3.patch, 3862_v8_addon.txt, include_memtables_in_rowcache_read.patch > > > While performing stress tests to find any race problems for CASSANDRA-2864 I > guess I (re-)found one for the standard on-heap row cache. > During my stress test I hava lots of threads running with some of them only > reading other writing and re-reading the value. > This seems to happen: > - Reader tries to read row A for the first time doing a getTopLevelColumns > - Row A which is not in the cache yet is updated by Writer. The row is not > eagerly read during write (because we want fast writes) so the writer cannot > perform a cache update > - Reader puts the row in the cache which is now missing the update > I already asked this some time ago on the mailing list but unfortunately > didn't dig after I got no answer since I assumed that I just missed > something. In a way I still do but haven't found any locking mechanism that > makes sure that this should not happen. > The problem can be reproduced with every run of my stress test. When I > restart the server the expected column is there. It's just missing from the > cache. > To test I have created a patch that merges memtables with the row cache. With > the patch the problem is gone. > I can also reproduce in 0.8. Haven't checked 1.1 but I haven't found any > relevant change their either so I assume the same aplies there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3763) compactionstats throws ArithmeticException: / by zero
[ https://issues.apache.org/jira/browse/CASSANDRA-3763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218417#comment-13218417 ] Zenek Kraweznik commented on CASSANDRA-3763: this bug may be linked with CASSANDRA-3693 > compactionstats throws ArithmeticException: / by zero > - > > Key: CASSANDRA-3763 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3763 > Project: Cassandra > Issue Type: Bug > Components: Core, Tools >Affects Versions: 1.0.7, 1.0.8 > Environment: debian linux - openvz kernel, oracle java 1.6.0.26 >Reporter: Zenek Kraweznik >Priority: Trivial > > compactionstats looks like this: > # nodetool -h localhost compactionstats > Exception in thread "main" java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435) > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128) > at > org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600) > at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > # > nodetool is working fine in other actions: > # nodetool -h localhost netstats > Mode: NORMAL > Not sending any streams. > Not receiving any streams. > Pool NameActive Pending Completed > Commandsn/a 0 2 > Responses n/a 0 1810 > # -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2261) During Compaction, Corrupt SSTables with rows that cause failures should be identified and blacklisted.
[ https://issues.apache.org/jira/browse/CASSANDRA-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Yaskevich updated CASSANDRA-2261: --- Attachment: CASSANDRA-2261-v3.patch changes in v3: # LeveledManifest now returns list with suspected SSTables filtered out if original candidates happen to have at least one suspected SSTables (originally it was just aborting compaction by returning empty list). # Blacklisting test now supportes both SizeTiered and Leveled strategies. Also for(;;) replaced with bounded loop to avoid the situation when we can get into infinite loop if forceMajorCompaction does not fail. # Styling and nits is fixed. > During Compaction, Corrupt SSTables with rows that cause failures should be > identified and blacklisted. > --- > > Key: CASSANDRA-2261 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2261 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benjamin Coverston >Assignee: Pavel Yaskevich >Priority: Minor > Labels: not_a_pony > Fix For: 1.1.1 > > Attachments: 2261-v2.patch, 2261.patch, CASSANDRA-2261-v3.patch > > > When a compaction of a set of SSTables fails because of corruption it will > continue to try to compact that SSTable causing pending compactions to build > up. > One way to mitigate this problem would be to log the error, then identify the > specific SSTable that caused the failure, subsequently blacklisting that > SSTable and ensuring that it is no longer included in future compactions. For > this we could simply store the problematic SSTable's name in memory. > If it's not possible to identify the SSTable that caused the issue, then > perhaps blacklisting the (ordered) permutation of SSTables to be compacted > together is something that can be done to solve this problem in a more > general case, and avoid issues where two (or more) SSTables have trouble > compacting a particular row. For this option we would probably want to store > the lists of the bad combinations in the system table somewhere s.t. these > can survive a node failure (there have been a few cases where I have seen a > compaction cause a node failure). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2261) During Compaction, Corrupt SSTables with rows that cause failures should be identified and blacklisted.
[ https://issues.apache.org/jira/browse/CASSANDRA-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218539#comment-13218539 ] Pavel Yaskevich edited comment on CASSANDRA-2261 at 2/28/12 7:53 PM: - changes in v3: - LeveledManifest now returns list with suspected SSTables filtered out if original candidates happen to have at least one suspected SSTables (originally it was just aborting compaction by returning empty list). - Blacklisting test now supportes both SizeTiered and Leveled strategies. Also for(;;) replaced with bounded loop to avoid the situation when we can get into infinite loop if forceMajorCompaction does not fail. - Styling and nits is fixed. was (Author: xedin): changes in v3: # LeveledManifest now returns list with suspected SSTables filtered out if original candidates happen to have at least one suspected SSTables (originally it was just aborting compaction by returning empty list). # Blacklisting test now supportes both SizeTiered and Leveled strategies. Also for(;;) replaced with bounded loop to avoid the situation when we can get into infinite loop if forceMajorCompaction does not fail. # Styling and nits is fixed. > During Compaction, Corrupt SSTables with rows that cause failures should be > identified and blacklisted. > --- > > Key: CASSANDRA-2261 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2261 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benjamin Coverston >Assignee: Pavel Yaskevich >Priority: Minor > Labels: not_a_pony > Fix For: 1.1.1 > > Attachments: 2261-v2.patch, 2261.patch, CASSANDRA-2261-v3.patch > > > When a compaction of a set of SSTables fails because of corruption it will > continue to try to compact that SSTable causing pending compactions to build > up. > One way to mitigate this problem would be to log the error, then identify the > specific SSTable that caused the failure, subsequently blacklisting that > SSTable and ensuring that it is no longer included in future compactions. For > this we could simply store the problematic SSTable's name in memory. > If it's not possible to identify the SSTable that caused the issue, then > perhaps blacklisting the (ordered) permutation of SSTables to be compacted > together is something that can be done to solve this problem in a more > general case, and avoid issues where two (or more) SSTables have trouble > compacting a particular row. For this option we would probably want to store > the lists of the bad combinations in the system table somewhere s.t. these > can survive a node failure (there have been a few cases where I have seen a > compaction cause a node failure). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2261) During Compaction, Corrupt SSTables with rows that cause failures should be identified and blacklisted.
[ https://issues.apache.org/jira/browse/CASSANDRA-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218564#comment-13218564 ] Brandon Williams commented on CASSANDRA-2261: - +1 > During Compaction, Corrupt SSTables with rows that cause failures should be > identified and blacklisted. > --- > > Key: CASSANDRA-2261 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2261 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benjamin Coverston >Assignee: Pavel Yaskevich >Priority: Minor > Labels: not_a_pony > Fix For: 1.1.1 > > Attachments: 2261-v2.patch, 2261.patch, CASSANDRA-2261-v3.patch > > > When a compaction of a set of SSTables fails because of corruption it will > continue to try to compact that SSTable causing pending compactions to build > up. > One way to mitigate this problem would be to log the error, then identify the > specific SSTable that caused the failure, subsequently blacklisting that > SSTable and ensuring that it is no longer included in future compactions. For > this we could simply store the problematic SSTable's name in memory. > If it's not possible to identify the SSTable that caused the issue, then > perhaps blacklisting the (ordered) permutation of SSTables to be compacted > together is something that can be done to solve this problem in a more > general case, and avoid issues where two (or more) SSTables have trouble > compacting a particular row. For this option we would probably want to store > the lists of the bad combinations in the system table somewhere s.t. these > can survive a node failure (there have been a few cases where I have seen a > compaction cause a node failure). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3684) Composite Column Support for PIG
[ https://issues.apache.org/jira/browse/CASSANDRA-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janne Jalkanen updated CASSANDRA-3684: -- Attachment: 3684-jalkanen.txt This patch (against cassandra-1.0) brings in basic support for Composite Columns. I needed a simple way to deconstruct an AbstractCompositeType, so I had to enhance that. Dunno if that's desireable, but it's certainly easy :-P Also the column name is an untyped tuple; not sure what would be the best way to extract the schema for it. > Composite Column Support for PIG > > > Key: CASSANDRA-3684 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3684 > Project: Cassandra > Issue Type: Bug >Reporter: Benjamin Coverston >Assignee: Brandon Williams > Attachments: 3684-jalkanen.txt > > > It appears that some changes need to be made to support CompositeColumns. > Right now if you try to load and use a column family that utilizes composite > columns you get the following exception[1]. > It appears to me that we need to modify the storage handler for Pig to > support this scenario. > [1] > > Backend error message > - > java.lang.RuntimeException: Unexpected data type -1 found in stream. > at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:478) > at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541) > at org.apache.pig.data.BinInterSedes.writeBag(BinInterSedes.java:522) > at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:361) > at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541) > at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:357) > at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:57) > at > org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:123) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1061) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691) > at > org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:239) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:272) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:266) > Backend error message > - > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) > Caused by: java.io.IOException: Task process exit with nonzero status of 65. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3684) Composite Column Support for PIG
[ https://issues.apache.org/jira/browse/CASSANDRA-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218570#comment-13218570 ] Janne Jalkanen commented on CASSANDRA-3684: --- Also, apologies for extra crap; my OCD demands that my git is configured to remove extra space at the end of the lines :). If this approach looks feasible, I'll make a cleaner patch. > Composite Column Support for PIG > > > Key: CASSANDRA-3684 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3684 > Project: Cassandra > Issue Type: Bug >Reporter: Benjamin Coverston >Assignee: Brandon Williams > Attachments: 3684-jalkanen.txt > > > It appears that some changes need to be made to support CompositeColumns. > Right now if you try to load and use a column family that utilizes composite > columns you get the following exception[1]. > It appears to me that we need to modify the storage handler for Pig to > support this scenario. > [1] > > Backend error message > - > java.lang.RuntimeException: Unexpected data type -1 found in stream. > at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:478) > at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541) > at org.apache.pig.data.BinInterSedes.writeBag(BinInterSedes.java:522) > at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:361) > at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:541) > at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:357) > at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:57) > at > org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:123) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90) > at > org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1061) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691) > at > org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:116) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:239) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:272) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:266) > Backend error message > - > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) > Caused by: java.io.IOException: Task process exit with nonzero status of 65. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3965) Cassandra code base has files with import statements having "*" causing compilation failure
[ https://issues.apache.org/jira/browse/CASSANDRA-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218581#comment-13218581 ] Dave Brosius commented on CASSANDRA-3965: - Yeah there wasn't one, it went in with https://issues.apache.org/jira/browse/CASSANDRA-3949 > Cassandra code base has files with import statements having "*" causing > compilation failure > --- > > Key: CASSANDRA-3965 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3965 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.0.8 > Environment: Linux >Reporter: Harish Doddi > > I tried to download a jar as part of the new unit test I am writing. I ran my > unit test successfully but later if I run "ant" without ant clean, I run into > the following compilation issue > {code} > build-project: > [echo] apache-cassandra: /Users/harish/workspace/cassandra/build.xml > [javac] Compiling 40 source files to > /Users/harish/workspace/cassandra/build/classes/thrift > [javac] Note: Some input files use unchecked or unsafe operations. > [javac] Note: Recompile with -Xlint:unchecked for details. > [javac] Compiling 568 source files to > /Users/harish/workspace/cassandra/build/classes/main > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/db/ColumnFamilyStore.java:1607: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Iterable> keySamples(Range > range) > [javac] ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:196: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Collection> getLocalRanges(String table) > [javac] ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:201: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Range getLocalPrimaryRange() > [javac]^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:912: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] public Map, List> > getRangeToAddressMap(String keyspace) > [javac]^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1009: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Map, List> > constructRangeToEndpointMap(String keyspace, List> ranges) > [javac] > ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1009: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Map, List> > constructRangeToEndpointMap(String keyspace, List> ranges) > [javac] ^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1438: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Multimap> > getNewSourceRanges(String table, Set> ranges) > [javac] >^ > [javac] > /Users/harish/workspace/cassandra/src/java/org/apache/cassandra/service/StorageService.java:1438: > reference to Range is ambiguous, both class org.apache.cassandra.dht.Range > in org.apache.cassandra.dht and class com.google.common.collect.Range in > com.google.common.collect match > [javac] private Multimap> > getNewSourceRanges(String table, Set> ranges) > [javac] ^ > [
git commit: identify and blacklist corrupted SSTables from future compactions patch by Benjamin Coverston and Pavel Yaskevich; reviewed by Brandon Williams for CASSANDRA-2261
Updated Branches: refs/heads/cassandra-1.1 2cc612ca0 -> 9483e147e identify and blacklist corrupted SSTables from future compactions patch by Benjamin Coverston and Pavel Yaskevich; reviewed by Brandon Williams for CASSANDRA-2261 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9483e147 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9483e147 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9483e147 Branch: refs/heads/cassandra-1.1 Commit: 9483e147ec675c6024b5f1a55c799cc15aa2239e Parents: 2cc612c Author: Pavel Yaskevich Authored: Tue Feb 28 22:45:24 2012 +0300 Committer: Pavel Yaskevich Committed: Tue Feb 28 23:37:45 2012 +0300 -- CHANGES.txt|2 +- .../db/columniterator/IndexedSliceReader.java |1 + .../db/columniterator/SSTableNamesIterator.java|2 + .../db/columniterator/SSTableSliceIterator.java|1 + .../db/columniterator/SimpleSliceReader.java |1 + .../db/compaction/AbstractCompactionStrategy.java | 24 +++- .../cassandra/db/compaction/LeveledManifest.java | 28 - .../compaction/SizeTieredCompactionStrategy.java |5 +- .../io/sstable/SSTableBoundedScanner.java |1 + .../io/sstable/SSTableIdentityIterator.java|1 + .../apache/cassandra/io/sstable/SSTableReader.java | 15 ++ .../cassandra/io/sstable/SSTableScanner.java |5 + .../cassandra/db/compaction/CompactionsTest.java | 113 ++- 13 files changed, 191 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 6a2a56e..591ae9f 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,6 @@ 1.1.1-dev * optimize commitlog checksumming (CASSANDRA-3610) - + * identify and blacklist corrupted SSTables from future compactions (CASSANDRA-2261) 1.1-dev * start hint replay as soon as FD notifies that the target is back up http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java -- diff --git a/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java b/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java index d2d9e9b..68c9a4c 100644 --- a/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java +++ b/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java @@ -76,6 +76,7 @@ class IndexedSliceReader extends AbstractIterator implements IColumnIte } catch (IOException e) { +sstable.markSuspect(); throw new IOError(e); } } http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java -- diff --git a/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java b/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java index fdb0c5c..ac8938c 100644 --- a/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java +++ b/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java @@ -73,6 +73,7 @@ public class SSTableNamesIterator extends SimpleAbstractColumnIterator implement } catch (IOException e) { +sstable.markSuspect(); throw new IOError(e); } finally @@ -93,6 +94,7 @@ public class SSTableNamesIterator extends SimpleAbstractColumnIterator implement } catch (IOException ioe) { +sstable.markSuspect(); throw new IOError(ioe); } } http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java -- diff --git a/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java b/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java index 72fe1bf..5c97729 100644 --- a/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java +++ b/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java @@ -60,6 +60,7 @@ public class SSTableSliceIterator implements IColumnIterator } catch (IOException e) { +sstable.markSuspect(); throw new IOError(e); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columnit
[2/2] git commit: identify and blacklist corrupted SSTables from future compactions patch by Benjamin Coverston and Pavel Yaskevich; reviewed by Brandon Williams for CASSANDRA-2261
identify and blacklist corrupted SSTables from future compactions patch by Benjamin Coverston and Pavel Yaskevich; reviewed by Brandon Williams for CASSANDRA-2261 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9483e147 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9483e147 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9483e147 Branch: refs/heads/trunk Commit: 9483e147ec675c6024b5f1a55c799cc15aa2239e Parents: 2cc612c Author: Pavel Yaskevich Authored: Tue Feb 28 22:45:24 2012 +0300 Committer: Pavel Yaskevich Committed: Tue Feb 28 23:37:45 2012 +0300 -- CHANGES.txt|2 +- .../db/columniterator/IndexedSliceReader.java |1 + .../db/columniterator/SSTableNamesIterator.java|2 + .../db/columniterator/SSTableSliceIterator.java|1 + .../db/columniterator/SimpleSliceReader.java |1 + .../db/compaction/AbstractCompactionStrategy.java | 24 +++- .../cassandra/db/compaction/LeveledManifest.java | 28 - .../compaction/SizeTieredCompactionStrategy.java |5 +- .../io/sstable/SSTableBoundedScanner.java |1 + .../io/sstable/SSTableIdentityIterator.java|1 + .../apache/cassandra/io/sstable/SSTableReader.java | 15 ++ .../cassandra/io/sstable/SSTableScanner.java |5 + .../cassandra/db/compaction/CompactionsTest.java | 113 ++- 13 files changed, 191 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 6a2a56e..591ae9f 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,6 +1,6 @@ 1.1.1-dev * optimize commitlog checksumming (CASSANDRA-3610) - + * identify and blacklist corrupted SSTables from future compactions (CASSANDRA-2261) 1.1-dev * start hint replay as soon as FD notifies that the target is back up http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java -- diff --git a/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java b/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java index d2d9e9b..68c9a4c 100644 --- a/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java +++ b/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java @@ -76,6 +76,7 @@ class IndexedSliceReader extends AbstractIterator implements IColumnIte } catch (IOException e) { +sstable.markSuspect(); throw new IOError(e); } } http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java -- diff --git a/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java b/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java index fdb0c5c..ac8938c 100644 --- a/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java +++ b/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java @@ -73,6 +73,7 @@ public class SSTableNamesIterator extends SimpleAbstractColumnIterator implement } catch (IOException e) { +sstable.markSuspect(); throw new IOError(e); } finally @@ -93,6 +94,7 @@ public class SSTableNamesIterator extends SimpleAbstractColumnIterator implement } catch (IOException ioe) { +sstable.markSuspect(); throw new IOError(ioe); } } http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java -- diff --git a/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java b/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java index 72fe1bf..5c97729 100644 --- a/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java +++ b/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java @@ -60,6 +60,7 @@ public class SSTableSliceIterator implements IColumnIterator } catch (IOException e) { +sstable.markSuspect(); throw new IOError(e); } http://git-wip-us.apache.org/repos/asf/cassandra/blob/9483e147/src/java/org/apache/cassandra/db/columniterator/SimpleSliceReader.java
[1/2] git commit: merge from 1.1
Updated Branches: refs/heads/trunk 5c7ccb033 -> be7cbc8dd merge from 1.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/be7cbc8d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/be7cbc8d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/be7cbc8d Branch: refs/heads/trunk Commit: be7cbc8dd21ab9b7c75405a0584e9a6b6655582e Parents: 5c7ccb0 9483e14 Author: Pavel Yaskevich Authored: Tue Feb 28 23:44:47 2012 +0300 Committer: Pavel Yaskevich Committed: Tue Feb 28 23:44:47 2012 +0300 -- CHANGES.txt|2 +- .../db/columniterator/IndexedSliceReader.java |1 + .../db/columniterator/SSTableNamesIterator.java|2 + .../db/columniterator/SSTableSliceIterator.java|1 + .../db/columniterator/SimpleSliceReader.java |1 + .../db/compaction/AbstractCompactionStrategy.java | 24 +++- .../cassandra/db/compaction/LeveledManifest.java | 28 - .../compaction/SizeTieredCompactionStrategy.java |5 +- .../io/sstable/SSTableBoundedScanner.java |1 + .../io/sstable/SSTableIdentityIterator.java|1 + .../apache/cassandra/io/sstable/SSTableReader.java | 15 ++ .../cassandra/io/sstable/SSTableScanner.java |5 + .../cassandra/db/compaction/CompactionsTest.java | 113 ++- 13 files changed, 191 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/db/columniterator/IndexedSliceReader.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/db/columniterator/SSTableNamesIterator.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/db/columniterator/SSTableSliceIterator.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/db/columniterator/SimpleSliceReader.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java -- diff --cc src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java index bb84d1d,2e70c46..5e0a911 --- a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java +++ b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java @@@ -15,11 -15,10 +15,9 @@@ * See the License for the specific language governing permissions and * limitations under the License. */ - package org.apache.cassandra.db.compaction; - import java.util.Collection; - import java.util.Map; - import java.util.Set; + import java.util.*; import java.util.concurrent.TimeUnit; import org.apache.cassandra.db.ColumnFamilyStore; http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/io/sstable/SSTableBoundedScanner.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/io/sstable/SSTableReader.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/be7cbc8d/src/java/org/apache/cassandra/io/sstable/SSTableScanner.java --
[jira] [Commented] (CASSANDRA-3710) Add a configuration option to disable snapshots
[ https://issues.apache.org/jira/browse/CASSANDRA-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218693#comment-13218693 ] Chris Herron commented on CASSANDRA-3710: - Adding notes from discussion on #cassandra-dev: There are legitimate production use cases that would benefit from being able to skip snapshots, both for truncate *and* dropColumnFamily. For example, our application has some ColumnFamilies that contain important data, and others that contain large volumes of derived data. Relatively often, we have the need to discard the derived data in order to recompute it. Truncate without snapshot would be ideal for this. While we could proactively prune snapshots, the coordination of such housekeeping out-of-process is complex. Differentiating "important" snapshots from unwanted snapshots when pruning requires that this process has knowledge of our domain model. Brandon stressed that the current philosophy is that Cassandra should never truly delete data, as a protection against accidental data loss. My own opinion is that Cassandra and its API should not work against the caller's intent. A compromise that we talked about is as follows: 1. Additional methods for truncate and dropColumnFamily that include a snapshot control flag, e.g: truncate(String keySpace, String cf, boolean withSnapshot) dropColumnFamily(String keySpace, String cf, boolean withSnapshot) 2. Add a "safety" configuration option that governs whether the snapshot flag is actually honored. So if(!withSnapshot && !allowSkippedSnapshotOnDeletion), we could either fail hard for an illegal request or log a warning and snapshot anyway. The advantage of this approach is that it would be equally useful for testing and production. The disadvantage is that it is an API change and so would have to come in a later release. An interim solution might be to add the configuration option (e.g. allowSkippedSnapshotOnDeletion, default false) and have truncate and delete honor that value directly. In a later release, the "withSnapshot" API additions could use allowSkippedSnapshotOnDeletion as described above in #2. > Add a configuration option to disable snapshots > --- > > Key: CASSANDRA-3710 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3710 > Project: Cassandra > Issue Type: New Feature >Reporter: Brandon Williams > Fix For: 1.0.9 > > Attachments: Cassandra107Patch_TestModeV1.txt > > > Let me first say, I hate this idea. It gives cassandra the ability to > permanently delete data at a large scale without any means of recovery. > However, I've seen this requested multiple times, and it is in fact useful in > some scenarios, such as when your application is using an embedded cassandra > instance for testing and need to truncate, which without JNA will timeout > more often than not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3975) Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily
[ https://issues.apache.org/jira/browse/CASSANDRA-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218695#comment-13218695 ] Jonathan Ellis commented on CASSANDRA-3975: --- Patch posted to https://github.com/jbellis/cassandra/branches/3975 > Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily > -- > > Key: CASSANDRA-3975 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3975 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.0 >Reporter: Chris Herron > Labels: datastax_qa > Fix For: 1.0.9, 1.1.0 > > > If hints have accumulated for a CF that has been deleted, Hinted Handoff > repeatedly fails until manual intervention removes those hints. For 1.0.7, > UnserializableColumnFamilyException is thrown only when a CFid is unknown on > the sending node. As discussed on #cassandra-dev, if the schema is in > agreement, the affected hint(s) should be deleted to avoid indefinite repeat > failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3442) TTL histogram for sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218762#comment-13218762 ] Yuki Morishita commented on CASSANDRA-3442: --- {quote} I switched from checking instanceof ExipiringColumn, to instanceof DeletedColumn, ... {quote} I don't see this switch in your patch, but as you described, tracking tombstones makes more sense. I think it's doable to create histogram of dropping time when writing sstable and use that for single sstable compaction. So we can trigger compaction on single sstable which contains say 20% of ExpiringColumns and DeletedColumns and 50% of them can be dropped. {quote} Minor note: the new test seems fairly involved – what would we lose by just testing compaction of a single sstable w/ tombstones? {quote} Well, nothing :) single sstable compaction test is fine. > TTL histogram for sstable metadata > -- > > Key: CASSANDRA-3442 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3442 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jonathan Ellis >Assignee: Yuki Morishita >Priority: Minor > Labels: compaction > Fix For: 1.2 > > Attachments: 3442-v3.txt, cassandra-1.1-3442.txt > > > Under size-tiered compaction, you can generate large sstables that compact > infrequently. With expiring columns mixed in, we could waste a lot of space > in this situation. > If we kept a TTL EstimatedHistogram in the sstable metadata, we could do a > single-sstable compaction aginst sstables with over 20% (?) expired data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3975) Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily
[ https://issues.apache.org/jira/browse/CASSANDRA-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218763#comment-13218763 ] Brandon Williams commented on CASSANDRA-3975: - +1 > Hints Should Be Dropped When Missing CFid Implies Deleted ColumnFamily > -- > > Key: CASSANDRA-3975 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3975 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.0.0 >Reporter: Chris Herron >Assignee: Jonathan Ellis > Labels: datastax_qa > Fix For: 1.0.9, 1.1.0 > > > If hints have accumulated for a CF that has been deleted, Hinted Handoff > repeatedly fails until manual intervention removes those hints. For 1.0.7, > UnserializableColumnFamilyException is thrown only when a CFid is unknown on > the sending node. As discussed on #cassandra-dev, if the schema is in > agreement, the affected hint(s) should be deleted to avoid indefinite repeat > failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3976) [patch[ don't compare byte arrays with ==
[ https://issues.apache.org/jira/browse/CASSANDRA-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Brosius updated CASSANDRA-3976: Attachment: cmp_bytearrays_w_equals.diff > [patch[ don't compare byte arrays with == > - > > Key: CASSANDRA-3976 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3976 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop >Reporter: Dave Brosius >Priority: Trivial > Attachments: cmp_bytearrays_w_equals.diff > > > code compares byte arrays with ==, use Arrays.equals > patch against trunk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3976) [patch[ don't compare byte arrays with ==
[patch[ don't compare byte arrays with == - Key: CASSANDRA-3976 URL: https://issues.apache.org/jira/browse/CASSANDRA-3976 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Dave Brosius Priority: Trivial Attachments: cmp_bytearrays_w_equals.diff code compares byte arrays with ==, use Arrays.equals patch against trunk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3976) [patch[ don't compare byte arrays with ==
[ https://issues.apache.org/jira/browse/CASSANDRA-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218865#comment-13218865 ] Jonathan Ellis commented on CASSANDRA-3976: --- Wouldn't just start/finish.length == 0 be simpler? > [patch[ don't compare byte arrays with == > - > > Key: CASSANDRA-3976 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3976 > Project: Cassandra > Issue Type: Improvement > Components: Hadoop >Reporter: Dave Brosius >Priority: Trivial > Attachments: cmp_bytearrays_w_equals.diff > > > code compares byte arrays with ==, use Arrays.equals > patch against trunk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[2/10] git commit: Merge branch 'cassandra-1.1.0' into cassandra-1.1
Merge branch 'cassandra-1.1.0' into cassandra-1.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/087aa315 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/087aa315 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/087aa315 Branch: refs/heads/cassandra-1.1 Commit: 087aa3158a19d1e4119bf044ce4db827afede6d9 Parents: 9483e14 bd9e402 Author: Jonathan Ellis Authored: Tue Feb 28 21:15:21 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 21:15:21 2012 -0600 -- CHANGES.txt|2 + .../cassandra/db/ColumnFamilySerializer.java |2 +- .../apache/cassandra/db/HintedHandOffManager.java | 18 +++- .../cassandra/db/UnknownColumnFamilyException.java | 33 +++ .../db/UnserializableColumnFamilyException.java| 33 --- .../apache/cassandra/db/commitlog/CommitLog.java |2 +- 6 files changed, 52 insertions(+), 38 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/087aa315/CHANGES.txt -- diff --cc CHANGES.txt index 591ae9f,b6f1447..db87e4b --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -26,9 -21,10 +26,11 @@@ Merged from 1.0 * (cqlsh) look for cqlshlib relative to realpath (CASSANDRA-3767) * Fix short read protection (CASSANDRA-3934) * always compact away deleted hints immediately after handoff (CASSANDRA-3955) + * delete hints from dropped ColumnFamilies on handoff instead of +erroring out (CASSANDRA-3975) + 1.1-beta1 * add nodetool rebuild_index (CASSANDRA-3583) * add nodetool rangekeysample (CASSANDRA-2917) http://git-wip-us.apache.org/repos/asf/cassandra/blob/087aa315/src/java/org/apache/cassandra/db/commitlog/CommitLog.java --
[1/10] git commit: Merge branch 'cassandra-1.1' into trunk
Updated Branches: refs/heads/cassandra-1.0 4a49b9dcc -> 24a708040 refs/heads/cassandra-1.1 9483e147e -> 087aa3158 refs/heads/cassandra-1.1.0 4fcb56fa3 -> bd9e40215 refs/heads/trunk be7cbc8dd -> 17ee45930 Merge branch 'cassandra-1.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/17ee4593 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/17ee4593 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/17ee4593 Branch: refs/heads/trunk Commit: 17ee459309b61fb1f0783ffa95554e14e8dfd926 Parents: be7cbc8 087aa31 Author: Jonathan Ellis Authored: Tue Feb 28 21:15:41 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 21:15:41 2012 -0600 -- CHANGES.txt|2 + .../cassandra/db/ColumnFamilySerializer.java |2 +- .../apache/cassandra/db/HintedHandOffManager.java | 18 +++- .../cassandra/db/UnknownColumnFamilyException.java | 32 +++ .../db/UnserializableColumnFamilyException.java| 32 --- .../apache/cassandra/db/commitlog/CommitLog.java |2 +- 6 files changed, 51 insertions(+), 37 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/17ee4593/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/17ee4593/src/java/org/apache/cassandra/db/HintedHandOffManager.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/17ee4593/src/java/org/apache/cassandra/db/UnknownColumnFamilyException.java -- diff --cc src/java/org/apache/cassandra/db/UnknownColumnFamilyException.java index 000,7bea9e9..e1b2983 mode 00,100644..100644 --- a/src/java/org/apache/cassandra/db/UnknownColumnFamilyException.java +++ b/src/java/org/apache/cassandra/db/UnknownColumnFamilyException.java @@@ -1,0 -1,33 +1,32 @@@ -package org.apache.cassandra.db; - -import java.io.IOException; - -/** ++/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * ++ * ++ * http://www.apache.org/licenses/LICENSE-2.0 ++ * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ ++package org.apache.cassandra.db; ++ ++import java.io.IOException; + + + public class UnknownColumnFamilyException extends IOException + { + public final int cfId; + + public UnknownColumnFamilyException(String msg, int cfId) + { + super(msg); + this.cfId = cfId; + } + } http://git-wip-us.apache.org/repos/asf/cassandra/blob/17ee4593/src/java/org/apache/cassandra/db/commitlog/CommitLog.java -- diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLog.java index ef16c40,69dd0b1..5b3c7b0 --- a/src/java/org/apache/cassandra/db/commitlog/CommitLog.java +++ b/src/java/org/apache/cassandra/db/commitlog/CommitLog.java @@@ -266,9 -267,9 +266,9 @@@ public class CommitLog implements Commi { // assuming version here. We've gone to lengths to make sure what gets written to the CL is in // the current version. so do make sure the CL is drained prior to upgrading a node. -rm = RowMutation.serializer().deserialize(new DataInputStream(bufIn), MessagingService.version_, IColumnSerializer.Flag.LOCAL); +rm = RowMutation.serializer().deserialize(new DataInputStream(bufIn), MessagingService.current_version, IColumnSerializer.Flag.LOCAL); } - catch (UnserializableColumnFamilyException ex) + catch (UnknownColumnFamilyException ex) { AtomicInteger i = invalidMutations.get(ex.cfId); if (i == null)
[3/10] git commit: Merge branch 'cassandra-1.1.0' into cassandra-1.1
Merge branch 'cassandra-1.1.0' into cassandra-1.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/087aa315 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/087aa315 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/087aa315 Branch: refs/heads/trunk Commit: 087aa3158a19d1e4119bf044ce4db827afede6d9 Parents: 9483e14 bd9e402 Author: Jonathan Ellis Authored: Tue Feb 28 21:15:21 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 21:15:21 2012 -0600 -- CHANGES.txt|2 + .../cassandra/db/ColumnFamilySerializer.java |2 +- .../apache/cassandra/db/HintedHandOffManager.java | 18 +++- .../cassandra/db/UnknownColumnFamilyException.java | 33 +++ .../db/UnserializableColumnFamilyException.java| 33 --- .../apache/cassandra/db/commitlog/CommitLog.java |2 +- 6 files changed, 52 insertions(+), 38 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/087aa315/CHANGES.txt -- diff --cc CHANGES.txt index 591ae9f,b6f1447..db87e4b --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -26,9 -21,10 +26,11 @@@ Merged from 1.0 * (cqlsh) look for cqlshlib relative to realpath (CASSANDRA-3767) * Fix short read protection (CASSANDRA-3934) * always compact away deleted hints immediately after handoff (CASSANDRA-3955) + * delete hints from dropped ColumnFamilies on handoff instead of +erroring out (CASSANDRA-3975) + 1.1-beta1 * add nodetool rebuild_index (CASSANDRA-3583) * add nodetool rangekeysample (CASSANDRA-2917) http://git-wip-us.apache.org/repos/asf/cassandra/blob/087aa315/src/java/org/apache/cassandra/db/commitlog/CommitLog.java --
[5/10] git commit: merge from 1.0
merge from 1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd9e4021 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd9e4021 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd9e4021 Branch: refs/heads/cassandra-1.1.0 Commit: bd9e40215405fa7cd553ac88384754f9b5cd8521 Parents: 4fcb56f 24a7080 Author: Jonathan Ellis Authored: Tue Feb 28 21:15:08 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 21:15:08 2012 -0600 -- CHANGES.txt|2 + .../cassandra/db/ColumnFamilySerializer.java |2 +- .../apache/cassandra/db/HintedHandOffManager.java | 18 +++- .../cassandra/db/UnknownColumnFamilyException.java | 33 +++ .../db/UnserializableColumnFamilyException.java| 33 --- .../apache/cassandra/db/commitlog/CommitLog.java |2 +- 6 files changed, 52 insertions(+), 38 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd9e4021/CHANGES.txt -- diff --cc CHANGES.txt index b5b79a9,cb6c255..b6f1447 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,109 -1,9 +1,111 @@@ -1.0.9 +1.1-dev + * start hint replay as soon as FD notifies that the target is back up + (CASSANDRA-3958) + * avoid unproductive deserializing of cached rows during compaction + (CASSANDRA-3921) + * fix concurrency issues with CQL keyspace creation (CASSANDRA-3903) + * Show Effective Owership via Nodetool ring (CASSANDRA-3412) + * Update ORDER BY syntax for CQL3 (CASSANDRA-3925) + * Fix BulkRecordWriter to not throw NPE if reducer gets no map data from Hadoop (CASSANDRA-3944) + * Fix bug with counters in super columns (CASSANDRA-3821) + * Remove deprecated merge_shard_chance (CASSANDRA-3940) + * add a convenient way to reset a node's schema (CASSANDRA-2963) + * fix for intermittent SchemaDisagreementException (CASSANDRA-3884) + * ignore deprecated KsDef/CfDef/ColumnDef fields in native schema (CASSANDRA-3963) + * CLI to report when unsupported column_metadata pair was given (CASSANDRA-3959) + * reincarnate removed and deprecated KsDef/CfDef attributes (CASSANDRA-3953) + * Fix race between writes and read for cache (CASSANDRA-3862) +Merged from 1.0: + * remove the wait on hint future during write (CASSANDRA-3870) + * (cqlsh) ignore missing CfDef opts (CASSANDRA-3933) + * (cqlsh) look for cqlshlib relative to realpath (CASSANDRA-3767) + * Fix short read protection (CASSANDRA-3934) * always compact away deleted hints immediately after handoff (CASSANDRA-3955) + * delete hints from dropped ColumnFamilies on handoff instead of +erroring out (CASSANDRA-3975) +1.1-beta1 + * add nodetool rebuild_index (CASSANDRA-3583) + * add nodetool rangekeysample (CASSANDRA-2917) + * Fix streaming too much data during move operations (CASSANDRA-3639) + * Nodetool and CLI connect to localhost by default (CASSANDRA-3568) + * Reduce memory used by primary index sample (CASSANDRA-3743) + * (Hadoop) separate input/output configurations (CASSANDRA-3197, 3765) + * avoid returning internal Cassandra classes over JMX (CASSANDRA-2805) + * add row-level isolation via SnapTree (CASSANDRA-2893) + * Optimize key count estimation when opening sstable on startup + (CASSANDRA-2988) + * multi-dc replication optimization supporting CL > ONE (CASSANDRA-3577) + * add command to stop compactions (CASSANDRA-1740, 3566, 3582) + * multithreaded streaming (CASSANDRA-3494) + * removed in-tree redhat spec (CASSANDRA-3567) + * "defragment" rows for name-based queries under STCS, again (CASSANDRA-2503) + * Recycle commitlog segments for improved performance + (CASSANDRA-3411, 3543, 3557, 3615) + * update size-tiered compaction to prioritize small tiers (CASSANDRA-2407) + * add message expiration logic to OutboundTcpConnection (CASSANDRA-3005) + * off-heap cache to use sun.misc.Unsafe instead of JNA (CASSANDRA-3271) + * EACH_QUORUM is only supported for writes (CASSANDRA-3272) + * replace compactionlock use in schema migration by checking CFS.isValid + (CASSANDRA-3116) + * recognize that "SELECT first ... *" isn't really "SELECT *" (CASSANDRA-3445) + * Use faster bytes comparison (CASSANDRA-3434) + * Bulk loader is no longer a fat client, (HADOOP) bulk load output format + (CASSANDRA-3045) + * (Hadoop) add support for KeyRange.filter + * remove assumption that keys and token are in bijection + (CASSANDRA-1034, 3574, 3604) + * always remove endpoints from delevery queue in HH (CASSANDRA-3546) + * fix race between cf flush and its 2ndary indexes flush (CASSANDRA-3547) + * fix potential race in AES when a repair fails (CASSANDRA-3548) + * Remove columns shadowed by a deleted container even when we c
[4/10] git commit: merge from 1.0
merge from 1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd9e4021 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd9e4021 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd9e4021 Branch: refs/heads/trunk Commit: bd9e40215405fa7cd553ac88384754f9b5cd8521 Parents: 4fcb56f 24a7080 Author: Jonathan Ellis Authored: Tue Feb 28 21:15:08 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 21:15:08 2012 -0600 -- CHANGES.txt|2 + .../cassandra/db/ColumnFamilySerializer.java |2 +- .../apache/cassandra/db/HintedHandOffManager.java | 18 +++- .../cassandra/db/UnknownColumnFamilyException.java | 33 +++ .../db/UnserializableColumnFamilyException.java| 33 --- .../apache/cassandra/db/commitlog/CommitLog.java |2 +- 6 files changed, 52 insertions(+), 38 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd9e4021/CHANGES.txt -- diff --cc CHANGES.txt index b5b79a9,cb6c255..b6f1447 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,109 -1,9 +1,111 @@@ -1.0.9 +1.1-dev + * start hint replay as soon as FD notifies that the target is back up + (CASSANDRA-3958) + * avoid unproductive deserializing of cached rows during compaction + (CASSANDRA-3921) + * fix concurrency issues with CQL keyspace creation (CASSANDRA-3903) + * Show Effective Owership via Nodetool ring (CASSANDRA-3412) + * Update ORDER BY syntax for CQL3 (CASSANDRA-3925) + * Fix BulkRecordWriter to not throw NPE if reducer gets no map data from Hadoop (CASSANDRA-3944) + * Fix bug with counters in super columns (CASSANDRA-3821) + * Remove deprecated merge_shard_chance (CASSANDRA-3940) + * add a convenient way to reset a node's schema (CASSANDRA-2963) + * fix for intermittent SchemaDisagreementException (CASSANDRA-3884) + * ignore deprecated KsDef/CfDef/ColumnDef fields in native schema (CASSANDRA-3963) + * CLI to report when unsupported column_metadata pair was given (CASSANDRA-3959) + * reincarnate removed and deprecated KsDef/CfDef attributes (CASSANDRA-3953) + * Fix race between writes and read for cache (CASSANDRA-3862) +Merged from 1.0: + * remove the wait on hint future during write (CASSANDRA-3870) + * (cqlsh) ignore missing CfDef opts (CASSANDRA-3933) + * (cqlsh) look for cqlshlib relative to realpath (CASSANDRA-3767) + * Fix short read protection (CASSANDRA-3934) * always compact away deleted hints immediately after handoff (CASSANDRA-3955) + * delete hints from dropped ColumnFamilies on handoff instead of +erroring out (CASSANDRA-3975) +1.1-beta1 + * add nodetool rebuild_index (CASSANDRA-3583) + * add nodetool rangekeysample (CASSANDRA-2917) + * Fix streaming too much data during move operations (CASSANDRA-3639) + * Nodetool and CLI connect to localhost by default (CASSANDRA-3568) + * Reduce memory used by primary index sample (CASSANDRA-3743) + * (Hadoop) separate input/output configurations (CASSANDRA-3197, 3765) + * avoid returning internal Cassandra classes over JMX (CASSANDRA-2805) + * add row-level isolation via SnapTree (CASSANDRA-2893) + * Optimize key count estimation when opening sstable on startup + (CASSANDRA-2988) + * multi-dc replication optimization supporting CL > ONE (CASSANDRA-3577) + * add command to stop compactions (CASSANDRA-1740, 3566, 3582) + * multithreaded streaming (CASSANDRA-3494) + * removed in-tree redhat spec (CASSANDRA-3567) + * "defragment" rows for name-based queries under STCS, again (CASSANDRA-2503) + * Recycle commitlog segments for improved performance + (CASSANDRA-3411, 3543, 3557, 3615) + * update size-tiered compaction to prioritize small tiers (CASSANDRA-2407) + * add message expiration logic to OutboundTcpConnection (CASSANDRA-3005) + * off-heap cache to use sun.misc.Unsafe instead of JNA (CASSANDRA-3271) + * EACH_QUORUM is only supported for writes (CASSANDRA-3272) + * replace compactionlock use in schema migration by checking CFS.isValid + (CASSANDRA-3116) + * recognize that "SELECT first ... *" isn't really "SELECT *" (CASSANDRA-3445) + * Use faster bytes comparison (CASSANDRA-3434) + * Bulk loader is no longer a fat client, (HADOOP) bulk load output format + (CASSANDRA-3045) + * (Hadoop) add support for KeyRange.filter + * remove assumption that keys and token are in bijection + (CASSANDRA-1034, 3574, 3604) + * always remove endpoints from delevery queue in HH (CASSANDRA-3546) + * fix race between cf flush and its 2ndary indexes flush (CASSANDRA-3547) + * fix potential race in AES when a repair fails (CASSANDRA-3548) + * Remove columns shadowed by a deleted container even when we cannot purg
[9/10] git commit: delete hints from dropped ColumnFamilies on handoff instead of erroring out patch by jbellis; reviewed by brandonwilliams for CASSANDRA-3975
delete hints from dropped ColumnFamilies on handoff instead of erroring out patch by jbellis; reviewed by brandonwilliams for CASSANDRA-3975 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/24a70804 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/24a70804 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/24a70804 Branch: refs/heads/cassandra-1.1.0 Commit: 24a70804086a07409fef1303f7c53327661806bc Parents: 4a49b9d Author: Jonathan Ellis Authored: Tue Feb 28 15:05:28 2012 -0600 Committer: Jonathan Ellis Committed: Tue Feb 28 21:13:43 2012 -0600 -- CHANGES.txt|2 + .../cassandra/db/ColumnFamilySerializer.java |5 +-- .../apache/cassandra/db/HintedHandOffManager.java | 18 +++- .../cassandra/db/UnknownColumnFamilyException.java | 33 +++ .../db/UnserializableColumnFamilyException.java| 33 --- .../apache/cassandra/db/commitlog/CommitLog.java |2 +- 6 files changed, 52 insertions(+), 41 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/24a70804/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 31ae77d..cb6c255 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,5 +1,7 @@ 1.0.9 * always compact away deleted hints immediately after handoff (CASSANDRA-3955) + * delete hints from dropped ColumnFamilies on handoff instead of + erroring out (CASSANDRA-3975) 1.0.8 http://git-wip-us.apache.org/repos/asf/cassandra/blob/24a70804/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java -- diff --git a/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java b/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java index 7f11948..abfe76c 100644 --- a/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java +++ b/src/java/org/apache/cassandra/db/ColumnFamilySerializer.java @@ -24,17 +24,14 @@ package org.apache.cassandra.db; import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; -import java.io.RandomAccessFile; import java.util.Collection; import org.apache.cassandra.config.Schema; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.io.IColumnSerializer; import org.apache.cassandra.io.ISerializer; -import org.apache.cassandra.io.sstable.SSTableMetadata; public class ColumnFamilySerializer implements ISerializer { @@ -126,7 +123,7 @@ public class ColumnFamilySerializer implements ISerializer // create a ColumnFamily based on the cf id int cfId = dis.readInt(); if (Schema.instance.getCF(cfId) == null) -throw new UnserializableColumnFamilyException("Couldn't find cfId=" + cfId, cfId); +throw new UnknownColumnFamilyException("Couldn't find cfId=" + cfId, cfId); ColumnFamily cf = ColumnFamily.create(cfId, factory); deserializeFromSSTableNoColumns(cf, dis); deserializeColumns(dis, cf, flag); http://git-wip-us.apache.org/repos/asf/cassandra/blob/24a70804/src/java/org/apache/cassandra/db/HintedHandOffManager.java -- diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java b/src/java/org/apache/cassandra/db/HintedHandOffManager.java index ca416d7..a9f979f 100644 --- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java +++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java @@ -349,13 +349,25 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean assert keyColumn != null; assert mutationColumn != null; DataInputStream in = new DataInputStream(ByteBufferUtil.inputStream(mutationColumn.value())); -RowMutation rm = RowMutation.serializer().deserialize(in, ByteBufferUtil.toInt(versionColumn.value())); +RowMutation rm; +try +{ +rm = RowMutation.serializer().deserialize(in, ByteBufferUtil.toInt(versionColumn.value())); +} +catch (UnknownColumnFamilyException e) +{ +logger_.debug("Skipping delivery of hint for deleted columnfamily", e); +rm = null; +} try { -sendMutation(endpoint, rm); +if (rm != null) +{ +sendMutation(endpoint, rm); +rowsReplayed++; +} deleteHint(tokenBytes