[jira] [Created] (CASSANDRA-10467) QoS for Queries
Benjamin Coverston created CASSANDRA-10467: -- Summary: QoS for Queries Key: CASSANDRA-10467 URL: https://issues.apache.org/jira/browse/CASSANDRA-10467 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benjamin Coverston Background: As an OLTP system that is based on pipelined execution worker pools can be saturated with long(er) running calls. When the system is under stress, those long running calls can make requests that should be short lived requests take a much longer period of time. Introduce the concept of QoS into Cassandra for client queries. A few ideas: 1. Allow clients to specify a QoS to be sent to Cassandra from the driver as part of the protocol. 2. Allow different requests to be tagged based on some simple criteria (perhaps configured) (i.e. ALLOW FILTERING was part of the query) 3. QOS based on the LUN that is accessed (SSDs get higher QOS than the RAID5) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9231) Support Routing Key as part of Partition Key
[ https://issues.apache.org/jira/browse/CASSANDRA-9231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14586256#comment-14586256 ] Benjamin Coverston commented on CASSANDRA-9231: --- I'm also -1 on adding UDFs into the mix, just on the merits of losing the token aware routing from the client. A simple designation of a some of the partition keys as routing keys would serve the use cases I'm aware of. > Support Routing Key as part of Partition Key > > > Key: CASSANDRA-9231 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9231 > Project: Cassandra > Issue Type: Wish > Components: Core >Reporter: Matthias Broecheler > Fix For: 3.x > > > Provide support for sub-dividing the partition key into a routing key and a > non-routing key component. Currently, all columns that make up the partition > key of the primary key are also routing keys, i.e. they determine which nodes > store the data. This proposal would give the data modeler the ability to > designate only a subset of the columns that comprise the partition key to be > routing keys. The non-routing key columns of the partition key identify the > partition but are not used to determine where to store the data. > Consider the following example table definition: > CREATE TABLE foo ( > a int, > b int, > c int, > d int, > PRIMARY KEY (([a], b), c ) ); > (a,b) is the partition key, c is the clustering key, and d is just a column. > In addition, the square brackets identify the routing key as column a. This > means that only the value of column a is used to determine the node for data > placement (i.e. only the value of column a is murmur3 hashed to compute the > token). In addition, column b is needed to identify the partition but does > not influence the placement. > This has the benefit that all rows with the same routing key (but potentially > different non-routing key columns of the partition key) are stored on the > same node and that knowledge of such co-locality can be exploited by > applications build on top of Cassandra. > Currently, the only way to achieve co-locality is within a partition. > However, this approach has the limitations that: a) there are theoretical and > (more importantly) practical limitations on the size of a partition and b) > rows within a partition are ordered and an index is build to exploit such > ordering. For large partitions that overhead is significant if ordering isn't > needed. > In other words, routing keys afford a simple means to achieve scalable > node-level co-locality without ordering while clustering keys afford > page-level co-locality with ordering. As such, they address different > co-locality needs giving the data modeler the flexibility to choose what is > needed for their application. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9663) Support For varint Counter Type
Benjamin Coverston created CASSANDRA-9663: - Summary: Support For varint Counter Type Key: CASSANDRA-9663 URL: https://issues.apache.org/jira/browse/CASSANDRA-9663 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Benjamin Coverston While 64 bits is enough for most counter use cases, it's pretty easy to overflow when doing some of squares (for example). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-7315) Hadoop Compatibility Broken
Benjamin Coverston created CASSANDRA-7315: - Summary: Hadoop Compatibility Broken Key: CASSANDRA-7315 URL: https://issues.apache.org/jira/browse/CASSANDRA-7315 Project: Cassandra Issue Type: Bug Components: Core, Hadoop Environment: Cassandra Reporter: Benjamin Coverston Attachments: 0001-fix-hadoopcompat-issues.patch Some enhancements we did recently for the CqlRecordReader are using the job context object directly to get the configuration. This patch fixes those issues so that we maintain compatibility with old and new mapreduce implementations. Attached is a patch for the 2.0 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7340) CqlRecordWriter doesn't have user/password authentication
[ https://issues.apache.org/jira/browse/CASSANDRA-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14025302#comment-14025302 ] Benjamin Coverston commented on CASSANDRA-7340: --- patch lgtm > CqlRecordWriter doesn't have user/password authentication > - > > Key: CASSANDRA-7340 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7340 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Reporter: Alex Liu >Assignee: Alex Liu > Fix For: 2.0.9 > > Attachments: 7340.txt > > > CqlRecordWriter doesn't create Cassandra client based on > "cassandra.output.keyspace.username" > and "cassandra.output.keyspace.passwd" setting. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-8944) Add double_counter Type
Benjamin Coverston created CASSANDRA-8944: - Summary: Add double_counter Type Key: CASSANDRA-8944 URL: https://issues.apache.org/jira/browse/CASSANDRA-8944 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston Fix For: 3.1 A double_counter type similar to the integer counter type that currently exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8977) Potential DOS Exposure from Netty
Benjamin Coverston created CASSANDRA-8977: - Summary: Potential DOS Exposure from Netty Key: CASSANDRA-8977 URL: https://issues.apache.org/jira/browse/CASSANDRA-8977 Project: Cassandra Issue Type: Bug Components: Core Reporter: Benjamin Coverston CVE-2014-3488 : The SslHandler in Netty before 3.9.2 allows remote attackers to cause a denial of service (infinite loop and CPU consumption) via a crafted SSLv2Hello message. 2.0 currently uses Netty 3.6.6 Upgrading to 3.9.2+ will fix this issue, and patch a few other known vulnerabilities that we do not currently expose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5553) Deterministic load ordering for keyspaces
[ https://issues.apache.org/jira/browse/CASSANDRA-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13736065#comment-13736065 ] Benjamin Coverston commented on CASSANDRA-5553: --- No, we can close it I think. We ended up doing it differently. > Deterministic load ordering for keyspaces > - > > Key: CASSANDRA-5553 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5553 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Benjamin Coverston >Assignee: Benjamin Coverston > > I'm working on an extension to C* that would benefit from being able to load > column families in a particular order. Right now load order is determined by > a keyset which ends up being random. If we made this deterministic I could > get my keyspace loaded before the other CF needs it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5999) Modify PRSI interface and related callers to use the index method containing the updated key and cf.
Benjamin Coverston created CASSANDRA-5999: - Summary: Modify PRSI interface and related callers to use the index method containing the updated key and cf. Key: CASSANDRA-5999 URL: https://issues.apache.org/jira/browse/CASSANDRA-5999 Project: Cassandra Issue Type: Improvement Reporter: Benjamin Coverston Assignee: Benjamin Coverston Fix For: 1.2.10 This enhancement will allow for better support of custom indexes using CQL3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-6000) Enhance UntypedResultSet.Row by adding missing/additional helper methods
[ https://issues.apache.org/jira/browse/CASSANDRA-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-6000: -- Attachment: cassandra-6000.patch Add support for getting list composition, and column metadata. > Enhance UntypedResultSet.Row by adding missing/additional helper methods > > > Key: CASSANDRA-6000 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6000 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benjamin Coverston > Fix For: 1.2.10 > > Attachments: cassandra-6000.patch > > > Add missing getList method, in addition it would be nice to have a method > that lets us get the column specifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5999) Modify PRSI interface and related callers to use the index method containing the updated key and cf.
[ https://issues.apache.org/jira/browse/CASSANDRA-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5999: -- Attachment: cassandra-5999.patch > Modify PRSI interface and related callers to use the index method containing > the updated key and cf. > > > Key: CASSANDRA-5999 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5999 > Project: Cassandra > Issue Type: Improvement >Reporter: Benjamin Coverston >Assignee: Benjamin Coverston > Fix For: 1.2.10 > > Attachments: cassandra-5999.patch > > > This enhancement will allow for better support of custom indexes using CQL3. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-6000) Enhance UntypedResultSet.Row by adding missing/additional helper methods
Benjamin Coverston created CASSANDRA-6000: - Summary: Enhance UntypedResultSet.Row by adding missing/additional helper methods Key: CASSANDRA-6000 URL: https://issues.apache.org/jira/browse/CASSANDRA-6000 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benjamin Coverston Fix For: 1.2.10 Add missing getList method, in addition it would be nice to have a method that lets us get the column specifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-6018) Add option to encrypt commitlog
[ https://issues.apache.org/jira/browse/CASSANDRA-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769821#comment-13769821 ] Benjamin Coverston commented on CASSANDRA-6018: --- Perhaps what we really need here is a hook for a filter chain on the commit log. CASSANDRA-4148 would also need something like this. > Add option to encrypt commitlog > > > Key: CASSANDRA-6018 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6018 > Project: Cassandra > Issue Type: Task > Components: Core >Reporter: Jason Brown >Assignee: Jason Brown > Labels: commit_log, encryption, security > Fix For: 1.2.10, 2.0.1 > > > We are going to start using cassandra for a billing system, and while I can > encrypt sstables at rest (via Datastax Enterprise), commit logs are more or > less plain text. Thus, an attacker would be able to easily read, for example, > credit card numbers in the clear text commit log (if the calling app does not > encrypt the data itself before sending it to cassandra). > I want to allow the option of encrypting the commit logs, most likely > controlled by a property in the yaml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5351) Avoid repairing already-repaired data by default
[ https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773178#comment-13773178 ] Benjamin Coverston commented on CASSANDRA-5351: --- I think this is a good idea. In fact, triggering when the un-repaired size gets to be above some configured threshold means that repairs can be tuned for your environment. If we had a set of SSTables that are 'repaired', another 'pending repair', and another set of 'not yet scheduled for repair' (or queued for future repair) you could actually monitor your system to see if repairs are 'keeping up' with your data volumes. > Avoid repairing already-repaired data by default > > > Key: CASSANDRA-5351 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5351 > Project: Cassandra > Issue Type: Task > Components: Core >Reporter: Jonathan Ellis >Assignee: Lyuben Todorov > Labels: repair > Fix For: 2.1 > > > Repair has always built its merkle tree from all the data in a columnfamily, > which is guaranteed to work but is inefficient. > We can improve this by remembering which sstables have already been > successfully repaired, and only repairing sstables new since the last repair. > (This automatically makes CASSANDRA-3362 much less of a problem too.) > The tricky part is, compaction will (if not taught otherwise) mix repaired > data together with non-repaired. So we should segregate unrepaired sstables > from the repaired ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2264) Restore From Snapshot eg: nodetool restore [filename] | ([keyspace] [columnfamily])
[ https://issues.apache.org/jira/browse/CASSANDRA-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston resolved CASSANDRA-2264. --- Resolution: Not A Problem A lot has changed since I created this. It doesn't make much sense to pursue this today. > Restore From Snapshot eg: nodetool restore [filename] | ([keyspace] > [columnfamily]) > - > > Key: CASSANDRA-2264 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2264 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.2 >Reporter: Benjamin Coverston > > Store additional metadata in the SSTable including: > generated_by: (From memtable, compaction, or cleanup) > ancestors: A list of SSTableNames|MD5sum > When executed it will copy the ancestors from the snapshot directory to the > data directory. > If given Keyspace and ColumnFamily arguments it will attempt to restore all > SSTables in the Keyspace/Columnfamily on that node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5351) Avoid repairing already-repaired data by default
[ https://issues.apache.org/jira/browse/CASSANDRA-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13775569#comment-13775569 ] Benjamin Coverston commented on CASSANDRA-5351: --- Worst case scenario we hit double the number of SSTables (assuming that un-repaired has the same number of levels and a key for the query in each). Best case scenario you're not updating the same data, so it gets filtered out in the interval tree query. The interval tree doesn't really discriminate between levels, they're all in the same pool, you just get the set of intersected SSTables. > Avoid repairing already-repaired data by default > > > Key: CASSANDRA-5351 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5351 > Project: Cassandra > Issue Type: Task > Components: Core >Reporter: Jonathan Ellis >Assignee: Lyuben Todorov > Labels: repair > Fix For: 2.1 > > > Repair has always built its merkle tree from all the data in a columnfamily, > which is guaranteed to work but is inefficient. > We can improve this by remembering which sstables have already been > successfully repaired, and only repairing sstables new since the last repair. > (This automatically makes CASSANDRA-3362 much less of a problem too.) > The tricky part is, compaction will (if not taught otherwise) mix repaired > data together with non-repaired. So we should segregate unrepaired sstables > from the repaired ones. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-6135) Add beforeChange Notification to Gossiper State.
Benjamin Coverston created CASSANDRA-6135: - Summary: Add beforeChange Notification to Gossiper State. Key: CASSANDRA-6135 URL: https://issues.apache.org/jira/browse/CASSANDRA-6135 Project: Cassandra Issue Type: New Feature Reporter: Benjamin Coverston Attachments: 0001-New-Gossiper-notification-to-IEndpointStateChangeSub.patch We would like an internal notification to be fired before state changes happen so we can intercept them, and in some cases defer them. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (CASSANDRA-6135) Add beforeChange Notification to Gossiper State.
[ https://issues.apache.org/jira/browse/CASSANDRA-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-6135: -- Attachment: 0001-New-Gossiper-notification-to-IEndpointStateChangeSub.patch Attached is Sergio's patch for 1.2 > Add beforeChange Notification to Gossiper State. > > > Key: CASSANDRA-6135 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6135 > Project: Cassandra > Issue Type: New Feature >Reporter: Benjamin Coverston > Attachments: > 0001-New-Gossiper-notification-to-IEndpointStateChangeSub.patch > > > We would like an internal notification to be fired before state changes > happen so we can intercept them, and in some cases defer them. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863170#comment-13863170 ] Benjamin Coverston commented on CASSANDRA-5201: --- I'll submit the reporter impl upstream. Thanks [~dvryaboy]! > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Dave Brosius > Attachments: 5201_a.txt, hadoopCompat.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: hadoopcompat-trunk.patch Attached trunk-rebased patch. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Attachments: 5201_a.txt, hadoopCompat.patch, hadoopcompat-trunk.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: progressable-fix.patch Attaching a patch that will fix the issue with progressable when used on the jobcontext object. I've tested this with the latest HDP release, and it works with pig. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915996#comment-13915996 ] Benjamin Coverston edited comment on CASSANDRA-5201 at 2/28/14 4:58 PM: [~moliware], attaching a patch that will fix the issue with progressable when used on the jobcontext object. I've tested this with the latest HDP release, and it works with pig. [~brandon.williams] take a look. was (Author: bcoverston): Attaching a patch that will fix the issue with progressable when used on the jobcontext object. I've tested this with the latest HDP release, and it works with pig. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918725#comment-13918725 ] Benjamin Coverston commented on CASSANDRA-5201: --- [~hkropp] I agree that we need to do the second change, but I don't see a need to add deployment classifiers. Because we're using Hadoop-Compat (from EB) a single binary will work for both. There's a niggle right now with Progressable that I'm working on, but a single binary will work for new and legacy deployments. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919579#comment-13919579 ] Benjamin Coverston commented on CASSANDRA-5201: --- Take a look at the code, it works for Hadoop1 and Hadoop2 without recompliation, and without shipping two sets of dependencies. Basically it detects the current version of Hadoop that you're running and dynamically determines which TaskAttemptContext to use, the Class, or the Interface. There's no need to use deployment classifiers to solve this particular problem. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: progressable-wrapper.patch Attaching a patch that brings in hadoop-compat, and adds the progressable wrapper to it. I'm going to submit this upstream to the elephant bird project, so we should be able to remove this code and add the dependency in the future. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch, progressable-wrapper.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: (was: progressable-wrapper.patch) > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch, progressable-wrapper.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: progressable-wrapper.patch Updated progressable-wrapper.patch to conform to current namespaces. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch, progressable-wrapper.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: (was: progressable-wrapper.patch) > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: progressable-wrapper.patch I removed the hadoop-compat dependency, as maven couldn't resolve the dependencies, reverted to the old version. This won't stop our input/output formats from being compatible with past and future releases. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoopCompat.patch, > hadoopcompat-trunk.patch, progressable-fix.patch, progressable-wrapper.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: hadoop-compat-2.1-merge.patch Patch for 2.1 branch > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Benjamin Coverston > Fix For: 2.0.6 > > Attachments: 5201_a.txt, hadoop-compat-2.1-merge.patch, > hadoopCompat.patch, hadoopcompat-trunk.patch, progressable-fix.patch, > progressable-wrapper.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-4665) Add Sanity Checks for User Modifiable Settings
Benjamin Coverston created CASSANDRA-4665: - Summary: Add Sanity Checks for User Modifiable Settings Key: CASSANDRA-4665 URL: https://issues.apache.org/jira/browse/CASSANDRA-4665 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benjamin Coverston Priority: Trivial I've looked at a several environments that have had some problems that may have been more easily identified if there was some type of warning or error thrown if some of the user modifiable settings had some sanity check associated with it. In one instance I saw a cluster that was having memory issues. After looking at the YAML they had: index_interval: 32 I'm not sure why anyone would set this lower than the default. They didn't even know what the setting did. Perhaps if we issued a warning on startup in cases like this it could have been resolved more quickly and easily. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4745) Required field 'keyspace' was not present! when modifying a keyspace
Benjamin Coverston created CASSANDRA-4745:
-
Summary: Required field 'keyspace' was not present! when
modifying a keyspace
Key: CASSANDRA-4745
URL: https://issues.apache.org/jira/browse/CASSANDRA-4745
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.1.4
Reporter: Benjamin Coverston
cassandra-cli gets stuck after updating an existing keyspace.
To reproduce, create the KS/CF through cassandra-stress then update the
keyspace as shown below in the CLI.
notroot@datastax-training:~/cassandra$ bin/cassandra-cli
Connected to: "Test Cluster" on 127.0.0.1/9160
Welcome to Cassandra CLI version 1.1.4
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default@unknown] update keyspace Keyspace1
... with placement_strategy = 'SimpleStrategy'
... and strategy_options = {replication_factor:2};
Not authenticated to a working keyspace.
fa270c1d-7c85-3dfc-98b0-db0755f853f3
Waiting for schema agreement...
... schemas agree across the cluster
org.apache.thrift.protocol.TProtocolException: Required field 'keyspace' was
not present! Struct: describe_keyspace_args(keyspace:null)
[default@unknown]
[default@unknown] use Keyspace1;
Login failure. Did you specify 'keyspace', 'username' and 'password'?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4774) IndexOutOfBoundsException in org.apache.cassandra.gms.Gossiper.sendGossip
Benjamin Coverston created CASSANDRA-4774: - Summary: IndexOutOfBoundsException in org.apache.cassandra.gms.Gossiper.sendGossip Key: CASSANDRA-4774 URL: https://issues.apache.org/jira/browse/CASSANDRA-4774 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.0 Environment: Saw this when looking through some logs in version 1.0.0 system was under a lot of load. Reporter: Benjamin Coverston ERROR [GossipTasks:1] 2012-10-06 10:47:48,390 Gossiper.java (line 169) Gossip error java.lang.IndexOutOfBoundsException: Index: 13, Size: 5 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.cassandra.gms.Gossiper.sendGossip(Gossiper.java:541) at org.apache.cassandra.gms.Gossiper.doGossipToUnreachableMember(Gossiper.java:575) at org.apache.cassandra.gms.Gossiper.access$300(Gossiper.java:59) at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:141) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4784) Create separate sstables for each token range handled by a node
[ https://issues.apache.org/jira/browse/CASSANDRA-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston reassigned CASSANDRA-4784: - Assignee: Benjamin Coverston > Create separate sstables for each token range handled by a node > --- > > Key: CASSANDRA-4784 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4784 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: sankalp kohli >Assignee: Benjamin Coverston >Priority: Minor > Labels: perfomance > > Currently, each sstable has data for all the ranges that node is handling. If > we change that and rather have separate sstables for each range that node is > handling, it can lead to some improvements. > Improvements > 1) Node rebuild will be very fast as sstables can be directly copied over to > the bootstrapping node. It will minimize any application level logic. We can > directly use Linux native methods to transfer sstables without using CPU and > putting less pressure on the serving node. I think in theory it will be the > fastest way to transfer data. > 2) Backup can only transfer sstables for a node which belong to its primary > keyrange. > 3) ETL process can only copy one replica of data and will be much faster. > Changes: > We can split the writes into multiple memtables for each range it is > handling. The sstables being flushed from these can have details of which > range of data it is handling. > There will be no change I think for any reads as they work with interleaved > data anyway. But may be we can improve there as well? > Complexities: > The change does not look very complicated. I am not taking into account how > it will work when ranges are being changed for nodes. > Vnodes might make this work more complicated. We can also have a bit on each > sstable which says whether it is primary data or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4784) Create separate sstables for each token range handled by a node
[ https://issues.apache.org/jira/browse/CASSANDRA-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486081#comment-13486081 ] Benjamin Coverston commented on CASSANDRA-4784: --- I have a working implementation of this for STCS, one issue is it has the unfortunate (or fortunate) side effect of also partitioning up the SSTables for LCS as I put the implementation inside the CompactionTask making the currently (small) SSTables much smaller. I feel like this puts us at a crossroads: Should we create a completely partitioned data strategy for vnodes (a directory per vnode), or should we continue to mix the data files in a single data directory? L0 to L1 compactions become particularly hairy if we do that unless we first partition the L0 SSTables then subsequently compact the partitioned L0 with L1 for the vnode. > Create separate sstables for each token range handled by a node > --- > > Key: CASSANDRA-4784 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4784 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: sankalp kohli >Assignee: Benjamin Coverston >Priority: Minor > Labels: perfomance > > Currently, each sstable has data for all the ranges that node is handling. If > we change that and rather have separate sstables for each range that node is > handling, it can lead to some improvements. > Improvements > 1) Node rebuild will be very fast as sstables can be directly copied over to > the bootstrapping node. It will minimize any application level logic. We can > directly use Linux native methods to transfer sstables without using CPU and > putting less pressure on the serving node. I think in theory it will be the > fastest way to transfer data. > 2) Backup can only transfer sstables for a node which belong to its primary > keyrange. > 3) ETL process can only copy one replica of data and will be much faster. > Changes: > We can split the writes into multiple memtables for each range it is > handling. The sstables being flushed from these can have details of which > range of data it is handling. > There will be no change I think for any reads as they work with interleaved > data anyway. But may be we can improve there as well? > Complexities: > The change does not look very complicated. I am not taking into account how > it will work when ranges are being changed for nodes. > Vnodes might make this work more complicated. We can also have a bit on each > sstable which says whether it is primary data or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4784) Create separate sstables for each token range handled by a node
[ https://issues.apache.org/jira/browse/CASSANDRA-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-4784: -- Attachment: 4784.patch > Create separate sstables for each token range handled by a node > --- > > Key: CASSANDRA-4784 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4784 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.2.0 beta 1 >Reporter: sankalp kohli >Assignee: Benjamin Coverston >Priority: Minor > Labels: perfomance > Fix For: 1.3 > > Attachments: 4784.patch > > > Currently, each sstable has data for all the ranges that node is handling. If > we change that and rather have separate sstables for each range that node is > handling, it can lead to some improvements. > Improvements > 1) Node rebuild will be very fast as sstables can be directly copied over to > the bootstrapping node. It will minimize any application level logic. We can > directly use Linux native methods to transfer sstables without using CPU and > putting less pressure on the serving node. I think in theory it will be the > fastest way to transfer data. > 2) Backup can only transfer sstables for a node which belong to its primary > keyrange. > 3) ETL process can only copy one replica of data and will be much faster. > Changes: > We can split the writes into multiple memtables for each range it is > handling. The sstables being flushed from these can have details of which > range of data it is handling. > There will be no change I think for any reads as they work with interleaved > data anyway. But may be we can improve there as well? > Complexities: > The change does not look very complicated. I am not taking into account how > it will work when ranges are being changed for nodes. > Vnodes might make this work more complicated. We can also have a bit on each > sstable which says whether it is primary data or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4784) Create separate sstables for each token range handled by a node
[ https://issues.apache.org/jira/browse/CASSANDRA-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493381#comment-13493381 ] Benjamin Coverston commented on CASSANDRA-4784: --- Like this? https://github.com/bcoverston/apache-hosted-cassandra/branches/4784 > Create separate sstables for each token range handled by a node > --- > > Key: CASSANDRA-4784 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4784 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.2.0 beta 1 >Reporter: sankalp kohli >Assignee: Benjamin Coverston >Priority: Minor > Labels: perfomance > Fix For: 1.3 > > Attachments: 4784.patch > > > Currently, each sstable has data for all the ranges that node is handling. If > we change that and rather have separate sstables for each range that node is > handling, it can lead to some improvements. > Improvements > 1) Node rebuild will be very fast as sstables can be directly copied over to > the bootstrapping node. It will minimize any application level logic. We can > directly use Linux native methods to transfer sstables without using CPU and > putting less pressure on the serving node. I think in theory it will be the > fastest way to transfer data. > 2) Backup can only transfer sstables for a node which belong to its primary > keyrange. > 3) ETL process can only copy one replica of data and will be much faster. > Changes: > We can split the writes into multiple memtables for each range it is > handling. The sstables being flushed from these can have details of which > range of data it is handling. > There will be no change I think for any reads as they work with interleaved > data anyway. But may be we can improve there as well? > Complexities: > The change does not look very complicated. I am not taking into account how > it will work when ranges are being changed for nodes. > Vnodes might make this work more complicated. We can also have a bit on each > sstable which says whether it is primary data or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5484) Support custom secondary indexes in CQL
Benjamin Coverston created CASSANDRA-5484: - Summary: Support custom secondary indexes in CQL Key: CASSANDRA-5484 URL: https://issues.apache.org/jira/browse/CASSANDRA-5484 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.4 Reporter: Benjamin Coverston Through thrift users can add custom secondary indexes to the column metadata. The following syntax is used in PLSQL, and I think we could use something similar. CREATE INDEX ON (COLUMN) [INDEXTYPE IS () [PARAMETERS ([, ])] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5484) Support custom secondary indexes in CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5484: -- Description: Through thrift users can add custom secondary indexes to the column metadata. The following syntax is used in PLSQL, and I think we could use something similar. CREATE INDEX ON () [INDEXTYPE IS () [PARAMETERS ([, ])] was: Through thrift users can add custom secondary indexes to the column metadata. The following syntax is used in PLSQL, and I think we could use something similar. CREATE INDEX ON (COLUMN) [INDEXTYPE IS () [PARAMETERS ([, ])] > Support custom secondary indexes in CQL > --- > > Key: CASSANDRA-5484 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5484 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.4 >Reporter: Benjamin Coverston > > Through thrift users can add custom secondary indexes to the column metadata. > The following syntax is used in PLSQL, and I think we could use something > similar. > CREATE INDEX ON () [INDEXTYPE IS () > [PARAMETERS ([, ])] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5484) Support custom secondary indexes in CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633442#comment-13633442 ] Benjamin Coverston commented on CASSANDRA-5484: --- wfm > Support custom secondary indexes in CQL > --- > > Key: CASSANDRA-5484 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5484 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.2.4 >Reporter: Benjamin Coverston > > Through thrift users can add custom secondary indexes to the column metadata. > The following syntax is used in PLSQL, and I think we could use something > similar. > CREATE INDEX ON () [INDEXTYPE IS () > [PARAMETERS ([, ])] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5547) Multi-threaded scrub
Benjamin Coverston created CASSANDRA-5547: - Summary: Multi-threaded scrub Key: CASSANDRA-5547 URL: https://issues.apache.org/jira/browse/CASSANDRA-5547 Project: Cassandra Issue Type: Improvement Reporter: Benjamin Coverston Scrub (especially offline) could benefit from being multi-threaded, especially in the case where the SSTables are compressed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5553) Enforce load ordering for keyspaces
Benjamin Coverston created CASSANDRA-5553: - Summary: Enforce load ordering for keyspaces Key: CASSANDRA-5553 URL: https://issues.apache.org/jira/browse/CASSANDRA-5553 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Reporter: Benjamin Coverston Assignee: Benjamin Coverston I'm working on an extension to C* that would benefit from being able to load column families in a particular order. Right now load order is determined by a keyset which ends up being random. I would like to allow the user to define a custom comparator to define load order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5553) Deterministic load ordering for keyspaces
[ https://issues.apache.org/jira/browse/CASSANDRA-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5553: -- Summary: Deterministic load ordering for keyspaces (was: Enforce load ordering for keyspaces) > Deterministic load ordering for keyspaces > - > > Key: CASSANDRA-5553 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5553 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Benjamin Coverston >Assignee: Benjamin Coverston > > I'm working on an extension to C* that would benefit from being able to load > column families in a particular order. Right now load order is determined by > a keyset which ends up being random. > I would like to allow the user to define a custom comparator to define load > order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5553) Deterministic load ordering for keyspaces
[ https://issues.apache.org/jira/browse/CASSANDRA-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5553: -- Description: I'm working on an extension to C* that would benefit from being able to load column families in a particular order. Right now load order is determined by a keyset which ends up being random. If we made this deterministic I could get my keyspace loaded before the other CF needs it. (was: I'm working on an extension to C* that would benefit from being able to load column families in a particular order. Right now load order is determined by a keyset which ends up being random. I would like to allow the user to define a custom comparator to define load order.) > Deterministic load ordering for keyspaces > - > > Key: CASSANDRA-5553 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5553 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Benjamin Coverston >Assignee: Benjamin Coverston > > I'm working on an extension to C* that would benefit from being able to load > column families in a particular order. Right now load order is determined by > a keyset which ends up being random. If we made this deterministic I could > get my keyspace loaded before the other CF needs it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5614) W/O specified columns ASPCSI does not get notified of deletes
Benjamin Coverston created CASSANDRA-5614:
-
Summary: W/O specified columns ASPCSI does not get notified of
deletes
Key: CASSANDRA-5614
URL: https://issues.apache.org/jira/browse/CASSANDRA-5614
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.5, 2.0
Reporter: Benjamin Coverston
I'm working on a secondary index implementation based on the composite index
type.
AbstractSimplePerColumnSecondaryIndex.java#delete is not called when CQL delete
statements do not specify columns.
When I specify columns it is called. Pretty sure this is a bug.
Setup:
{code}
cqlsh> create KEYSPACE foo WITH replication = {'class': 'SimpleStrategy' ,
'replication_factor': 1};
cqlsh> use foo;
cqlsh:foo> CREATE TABLE albums (artist text, album text, rating int, release
int, PRIMARY KEY (artist, album));
cqlsh:foo> CREATE INDEX ON albums (rating);
{code}
{code}
cqlsh:foo> insert into albums (artist, album, rating, release) VALUES
('artist', 'album', 1, 2);
{code}
Does not get called here:
{code}
cqlsh:foo> DELETE FROM albums where artist='artist' and album='album';
{code}
{code}
cqlsh:foo> insert into albums (artist, album, rating, release) VALUES
('artist', 'album', 1, 2);
{code}
gets called here:
{code}
cqlsh:foo> DELETE rating FROM albums where artist='artist' and album='album';
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5611) OS X puts an icon for Cassandra in the dock
[
https://issues.apache.org/jira/browse/CASSANDRA-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1367#comment-1367
]
Benjamin Coverston commented on CASSANDRA-5611:
---
Nothing here:
{code}
Ben-Coverstons-MacBook-Pro:riptano-cassandra bcoverston$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01-451, mixed mode)
{code}
Also not getting anything after upgrading:
{code}
Ben-Coverstons-MacBook-Pro:riptano-cassandra bcoverston$ java -version
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b12)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
{code}
> OS X puts an icon for Cassandra in the dock
> ---
>
> Key: CASSANDRA-5611
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5611
> Project: Cassandra
> Issue Type: Improvement
>Affects Versions: 1.1.12, 1.2.5, 2.0
> Environment: OS X, JDK 1.7
>Reporter: Aleksey Yeschenko
>Priority: Trivial
>
> "Even when a Java program doesn't display any windows or other visible
> elements, if it accesses the AWT subsystem in some way (e.g., to do image
> processing internally), OS X will still put an icon for the Java program in
> the dock as if it were a GUI-based app. (When the program quits, the dock
> icon goes away as usual.)"
> (more details at
> http://hints.macworld.com/article.php?story=20071208235352641)
> Can't remember when it started happening, but it wasn't always the case for
> Cassandra. Now launching Cassandra puts an icon in the dock, and, worse,
> running ant test puts an icon in the dock for each test, stealing focus, too.
> This is extremely annoying.
> I ninja-d a workaround (-Djava.awt.headless=true) for ant test in
> 99824496aa359fcbe5e71f4e54f2738f09524a87, but we should try and find the real
> source of this (my guess is that some dependency of ours is to blame).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5615) Allow Custom Indexes to Index Collections
Benjamin Coverston created CASSANDRA-5615: - Summary: Allow Custom Indexes to Index Collections Key: CASSANDRA-5615 URL: https://issues.apache.org/jira/browse/CASSANDRA-5615 Project: Cassandra Issue Type: Improvement Affects Versions: 1.2.5 Reporter: Benjamin Coverston Creating a custom index type that will index collections. Attaching a patch that bypasses an internal validation check preventing this type of index for custom indexes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5615) Allow Custom Indexes to Index Collections
[ https://issues.apache.org/jira/browse/CASSANDRA-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5615: -- Attachment: custom-index-keys.patch Allows for custom indexes on collections > Allow Custom Indexes to Index Collections > - > > Key: CASSANDRA-5615 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5615 > Project: Cassandra > Issue Type: Improvement >Affects Versions: 1.2.5 >Reporter: Benjamin Coverston > Attachments: custom-index-keys.patch > > > Creating a custom index type that will index collections. Attaching a patch > that bypasses an internal validation check preventing this type of index for > custom indexes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5614) W/O specified columns ASPCSI does not get notified of deletes
[
https://issues.apache.org/jira/browse/CASSANDRA-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676057#comment-13676057
]
Benjamin Coverston commented on CASSANDRA-5614:
---
Instead of expanding them, at least for the purposes of what I'm doing I would
be OK with just getting keybytes, and column if column had a special marker for
a row tombstone.
> W/O specified columns ASPCSI does not get notified of deletes
> -
>
> Key: CASSANDRA-5614
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5614
> Project: Cassandra
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Benjamin Coverston
>Assignee: Sam Overton
>Priority: Minor
> Fix For: 1.2.6
>
>
> I'm working on a secondary index implementation based on the composite index
> type.
> AbstractSimplePerColumnSecondaryIndex.java#delete is not called when CQL
> delete statements do not specify columns.
> When I specify columns it is called. Pretty sure this is a bug.
> Setup:
> {code}
> cqlsh> create KEYSPACE foo WITH replication = {'class': 'SimpleStrategy' ,
> 'replication_factor': 1};
> cqlsh> use foo;
> cqlsh:foo> CREATE TABLE albums (artist text, album text, rating int, release
> int, PRIMARY KEY (artist, album));
> cqlsh:foo> CREATE INDEX ON albums (rating);
> {code}
> {code}
> cqlsh:foo> insert into albums (artist, album, rating, release) VALUES
> ('artist', 'album', 1, 2);
> {code}
> Does not get called here:
> {code}
> cqlsh:foo> DELETE FROM albums where artist='artist' and album='album';
> {code}
> {code}
> cqlsh:foo> insert into albums (artist, album, rating, release) VALUES
> ('artist', 'album', 1, 2);
> {code}
> gets called here:
> {code}
> cqlsh:foo> DELETE rating FROM albums where artist='artist' and album='album';
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-5614) W/O specified columns ASPCSI does not get notified of deletes
[
https://issues.apache.org/jira/browse/CASSANDRA-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676057#comment-13676057
]
Benjamin Coverston edited comment on CASSANDRA-5614 at 6/5/13 3:55 PM:
---
Instead of expanding them, at least for the purposes of what I'm doing, I would
be OK with just getting keybytes, and column if column had a special marker for
a row tombstone. That way I know the whole row was deleted up front (and I can
take one action) rather than getting a notification per column.
was (Author: bcoverston):
Instead of expanding them, at least for the purposes of what I'm doing I
would be OK with just getting keybytes, and column if column had a special
marker for a row tombstone.
> W/O specified columns ASPCSI does not get notified of deletes
> -
>
> Key: CASSANDRA-5614
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5614
> Project: Cassandra
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Benjamin Coverston
>Assignee: Sam Overton
>Priority: Minor
> Fix For: 1.2.6
>
>
> I'm working on a secondary index implementation based on the composite index
> type.
> AbstractSimplePerColumnSecondaryIndex.java#delete is not called when CQL
> delete statements do not specify columns.
> When I specify columns it is called. Pretty sure this is a bug.
> Setup:
> {code}
> cqlsh> create KEYSPACE foo WITH replication = {'class': 'SimpleStrategy' ,
> 'replication_factor': 1};
> cqlsh> use foo;
> cqlsh:foo> CREATE TABLE albums (artist text, album text, rating int, release
> int, PRIMARY KEY (artist, album));
> cqlsh:foo> CREATE INDEX ON albums (rating);
> {code}
> {code}
> cqlsh:foo> insert into albums (artist, album, rating, release) VALUES
> ('artist', 'album', 1, 2);
> {code}
> Does not get called here:
> {code}
> cqlsh:foo> DELETE FROM albums where artist='artist' and album='album';
> {code}
> {code}
> cqlsh:foo> insert into albums (artist, album, rating, release) VALUES
> ('artist', 'album', 1, 2);
> {code}
> gets called here:
> {code}
> cqlsh:foo> DELETE rating FROM albums where artist='artist' and album='album';
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-6219) Add cause for IOException in OutboundTCPConnectio.WriteConnected
[ https://issues.apache.org/jira/browse/CASSANDRA-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-6219: -- Attachment: 0001-added-cause-to-logging-message.patch > Add cause for IOException in OutboundTCPConnectio.WriteConnected > > > Key: CASSANDRA-6219 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6219 > Project: Cassandra > Issue Type: Bug >Reporter: Benjamin Coverston >Priority: Minor > Attachments: 0001-added-cause-to-logging-message.patch > > > Can help for debugging connection issues. Attaching patch for 1.2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (CASSANDRA-6219) Add cause for IOException in OutboundTCPConnectio.WriteConnected
Benjamin Coverston created CASSANDRA-6219: - Summary: Add cause for IOException in OutboundTCPConnectio.WriteConnected Key: CASSANDRA-6219 URL: https://issues.apache.org/jira/browse/CASSANDRA-6219 Project: Cassandra Issue Type: Bug Reporter: Benjamin Coverston Priority: Minor Can help for debugging connection issues. Attaching patch for 1.2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6219) Add cause for IOException in OutboundTCPConnectio.WriteConnected
[ https://issues.apache.org/jira/browse/CASSANDRA-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799664#comment-13799664 ] Benjamin Coverston commented on CASSANDRA-6219: --- I think you're right, reading the IOException code agrees with what you're saying. It _may_ not be in the stack trace, but it is in the detail message. > Add cause for IOException in OutboundTCPConnectio.WriteConnected > > > Key: CASSANDRA-6219 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6219 > Project: Cassandra > Issue Type: Bug >Reporter: Benjamin Coverston >Priority: Minor > Attachments: 0001-added-cause-to-logging-message.patch > > > Can help for debugging connection issues. Attaching patch for 1.2. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-5201: -- Attachment: hadoopCompat.patch Poking around at other projects this generally gets solved in one of two ways: Ship two versions of their Hadoop integrations (one compiled for the old, and one compiled for the new), or use a little reflection to make things work across the board. I'm attaching a patch that uses the hadoopCompat subproject of elephantbird. This will allow us to compile a single binary and run with the new and old context objects. I've tested this patch with HDP 2.0, and Apache Hadoop 1.0.4 and it works fine with both (including Hive in DSE). With Pig I needed to compile our (optional) pig dependency with: bq. ant clean jar-withouthadoop -Dhadoopversion=23 Only really needed if you're using one of the current versions of thrift with the new JobContext. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Dave Brosius > Attachments: 5201_a.txt, hadoopCompat.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5201) Cassandra/Hadoop does not support current Hadoop releases
[ https://issues.apache.org/jira/browse/CASSANDRA-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13838325#comment-13838325 ] Benjamin Coverston commented on CASSANDRA-5201: --- These changes also depend on CASSANDRA-6309 for anything to work. > Cassandra/Hadoop does not support current Hadoop releases > - > > Key: CASSANDRA-5201 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5201 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Affects Versions: 1.2.0 >Reporter: Brian Jeltema >Assignee: Dave Brosius > Attachments: 5201_a.txt, hadoopCompat.patch > > > Using Hadoop 0.22.0 with Cassandra results in the stack trace below. > It appears that version 0.21+ changed org.apache.hadoop.mapreduce.JobContext > from a class to an interface. > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-2699) continuous incremental anti-entropy
[
https://issues.apache.org/jira/browse/CASSANDRA-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583453#comment-13583453
]
Benjamin Coverston commented on CASSANDRA-2699:
---
>From CASSANDRA-4482
{quote}
Instead, we maintain a Merkle Tree (MT) in memory and update it with every
single column insert in ColumnFamilyStore.apply(). We use
column.updateDigest(digest) on all the changes in order to create a hash per
column update and then XOR this hash with the existing one in the Merkle Tree
bucket for the corresponding row.
This Merkle Tree is created with the column family (one per range), initialized
with zeros, and persisted to disk with regular snapshots.
The commutative properties of XOR make it possible to update the MT
incrementally without having to read on write.
{quote}
I'm pretty sure this means that we should be able to XOR the buckets together
from pre-computed merkle tree SSTable components.
We could just create these on flush, merge them on compaction, then validation
compaction is just a read, MT components and merge.
> continuous incremental anti-entropy
> ---
>
> Key: CASSANDRA-2699
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2699
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Peter Schuller
>Assignee: Peter Schuller
>
> Currently, repair works by periodically running "bulk" jobs that (1)
> performs a validating compaction building up an in-memory merkle tree,
> and (2) streaming ring segments as needed according to differences
> indicated by the merkle tree.
> There are some disadvantages to this approach:
> * There is a trade-off between memory usage and the precision of the
> merkle tree. Less precision means more data streamed relative to
> what is strictly required.
> * Repair is a periodic "bulk" process that runs for a significant
> period and, although possibly rate limited as compaction (if 0.8 or
> backported throttling patch applied), is a divergence in terms of
> performance characteristics from "normal" operation of the cluster.
> * The impact of imprecision can be huge on a workload dominated by I/O
> and with cache locality being critical, since you will suddenly
> transfers lots of data to the target node.
> I propose a more incremental process whereby anti-entropy is
> incremental and continuous over time. In order to avoid being
> seek-bound one still wants to do work in some form of bursty fashion,
> but the amount of data processed at a time could be sufficiently small
> that the impact on the cluster feels a lot more continuous, and that
> the page cache allows us to avoid re-reading differing data twice.
> Consider a process whereby a node is constantly performing a per-CF
> repair operation for each CF. The current state of the repair process
> is defined by:
> * A starting timestamp of the current iteration through the token
> range the node is responsible for.
> * A "finger" indicating the current position along the token ring to
> which iteration has completed.
> This information, other than being in-memory, could periodically (every
> few minutes or something) be stored persistently on disk.
> The finger advances by the node selecting the next small "bit" of the
> ring and doing whatever merkling/hashing/checksumming is necessary on
> that small part, and then asking neighbors to do the same, and
> arranging for neighbors to send the node data for mismatching
> ranges. The data would be sent either by way of mutations like with
> read repair, or by streaming sstables. But it would be small amounts
> of data that will act roughly the same as regular writes for the
> perspective of compaction.
> Some nice properties of this approach:
> * It's "always on"; no periodic sudden effects on cluster performance.
> * Restarting nodes never cancels or breaks anti-entropy.
> * Huge compactions of entire CF:s never clog up the compaction queue
> (not necessarily a non-issue even with concurrent compactions in
> 0.8).
> * Because we're always operating on small chunks, there is never the
> same kind of trade-off for memory use. A merkel tree or similar
> could be calculated at a very detailed level potentially. Although
> the precision from the perspective of reading from disk would likely
> not matter much if we are in page cache anyway, very high precision
> could be *very* useful when doing anti-entropy across data centers
> on slow links.
> There are devils in details, like how to select an appropriate ring
> segment given that you don't have knowledge of the data density on
> other nodes. But I feel that the overall idea/process seems very
> promising.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
Fo
[jira] [Comment Edited] (CASSANDRA-2699) continuous incremental anti-entropy
[
https://issues.apache.org/jira/browse/CASSANDRA-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583453#comment-13583453
]
Benjamin Coverston edited comment on CASSANDRA-2699 at 2/21/13 7:19 PM:
>From CASSANDRA-4482
{quote}
Instead, we maintain a Merkle Tree (MT) in memory and update it with every
single column insert in ColumnFamilyStore.apply(). We use
column.updateDigest(digest) on all the changes in order to create a hash per
column update and then XOR this hash with the existing one in the Merkle Tree
bucket for the corresponding row.
This Merkle Tree is created with the column family (one per range), initialized
with zeros, and persisted to disk with regular snapshots.
The commutative properties of XOR make it possible to update the MT
incrementally without having to read on write.
{quote}
I'm pretty sure this means that we should be able to XOR the buckets together
from pre-computed merkle tree SSTable components.
We could just create these on flush, merge them on compaction, then validation
compaction is just a read of MT components and merge.
was (Author: bcoverston):
From CASSANDRA-4482
{quote}
Instead, we maintain a Merkle Tree (MT) in memory and update it with every
single column insert in ColumnFamilyStore.apply(). We use
column.updateDigest(digest) on all the changes in order to create a hash per
column update and then XOR this hash with the existing one in the Merkle Tree
bucket for the corresponding row.
This Merkle Tree is created with the column family (one per range), initialized
with zeros, and persisted to disk with regular snapshots.
The commutative properties of XOR make it possible to update the MT
incrementally without having to read on write.
{quote}
I'm pretty sure this means that we should be able to XOR the buckets together
from pre-computed merkle tree SSTable components.
We could just create these on flush, merge them on compaction, then validation
compaction is just a read, MT components and merge.
> continuous incremental anti-entropy
> ---
>
> Key: CASSANDRA-2699
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2699
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Peter Schuller
>Assignee: Peter Schuller
>
> Currently, repair works by periodically running "bulk" jobs that (1)
> performs a validating compaction building up an in-memory merkle tree,
> and (2) streaming ring segments as needed according to differences
> indicated by the merkle tree.
> There are some disadvantages to this approach:
> * There is a trade-off between memory usage and the precision of the
> merkle tree. Less precision means more data streamed relative to
> what is strictly required.
> * Repair is a periodic "bulk" process that runs for a significant
> period and, although possibly rate limited as compaction (if 0.8 or
> backported throttling patch applied), is a divergence in terms of
> performance characteristics from "normal" operation of the cluster.
> * The impact of imprecision can be huge on a workload dominated by I/O
> and with cache locality being critical, since you will suddenly
> transfers lots of data to the target node.
> I propose a more incremental process whereby anti-entropy is
> incremental and continuous over time. In order to avoid being
> seek-bound one still wants to do work in some form of bursty fashion,
> but the amount of data processed at a time could be sufficiently small
> that the impact on the cluster feels a lot more continuous, and that
> the page cache allows us to avoid re-reading differing data twice.
> Consider a process whereby a node is constantly performing a per-CF
> repair operation for each CF. The current state of the repair process
> is defined by:
> * A starting timestamp of the current iteration through the token
> range the node is responsible for.
> * A "finger" indicating the current position along the token ring to
> which iteration has completed.
> This information, other than being in-memory, could periodically (every
> few minutes or something) be stored persistently on disk.
> The finger advances by the node selecting the next small "bit" of the
> ring and doing whatever merkling/hashing/checksumming is necessary on
> that small part, and then asking neighbors to do the same, and
> arranging for neighbors to send the node data for mismatching
> ranges. The data would be sent either by way of mutations like with
> read repair, or by streaming sstables. But it would be small amounts
> of data that will act roughly the same as regular writes for the
> perspective of compaction.
> Some nice properties of this approach:
> * It's "always on"; no periodic sudden effects on cluster performance.
> * Restarting n
[jira] [Commented] (CASSANDRA-2699) continuous incremental anti-entropy
[ https://issues.apache.org/jira/browse/CASSANDRA-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588347#comment-13588347 ] Benjamin Coverston commented on CASSANDRA-2699: --- You're right, which also means that in the face of idempotent writes and replay the incremental scenario is also broken with the in-memory tree. > continuous incremental anti-entropy > --- > > Key: CASSANDRA-2699 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2699 > Project: Cassandra > Issue Type: Improvement >Reporter: Peter Schuller >Assignee: Peter Schuller > > Currently, repair works by periodically running "bulk" jobs that (1) > performs a validating compaction building up an in-memory merkle tree, > and (2) streaming ring segments as needed according to differences > indicated by the merkle tree. > There are some disadvantages to this approach: > * There is a trade-off between memory usage and the precision of the > merkle tree. Less precision means more data streamed relative to > what is strictly required. > * Repair is a periodic "bulk" process that runs for a significant > period and, although possibly rate limited as compaction (if 0.8 or > backported throttling patch applied), is a divergence in terms of > performance characteristics from "normal" operation of the cluster. > * The impact of imprecision can be huge on a workload dominated by I/O > and with cache locality being critical, since you will suddenly > transfers lots of data to the target node. > I propose a more incremental process whereby anti-entropy is > incremental and continuous over time. In order to avoid being > seek-bound one still wants to do work in some form of bursty fashion, > but the amount of data processed at a time could be sufficiently small > that the impact on the cluster feels a lot more continuous, and that > the page cache allows us to avoid re-reading differing data twice. > Consider a process whereby a node is constantly performing a per-CF > repair operation for each CF. The current state of the repair process > is defined by: > * A starting timestamp of the current iteration through the token > range the node is responsible for. > * A "finger" indicating the current position along the token ring to > which iteration has completed. > This information, other than being in-memory, could periodically (every > few minutes or something) be stored persistently on disk. > The finger advances by the node selecting the next small "bit" of the > ring and doing whatever merkling/hashing/checksumming is necessary on > that small part, and then asking neighbors to do the same, and > arranging for neighbors to send the node data for mismatching > ranges. The data would be sent either by way of mutations like with > read repair, or by streaming sstables. But it would be small amounts > of data that will act roughly the same as regular writes for the > perspective of compaction. > Some nice properties of this approach: > * It's "always on"; no periodic sudden effects on cluster performance. > * Restarting nodes never cancels or breaks anti-entropy. > * Huge compactions of entire CF:s never clog up the compaction queue > (not necessarily a non-issue even with concurrent compactions in > 0.8). > * Because we're always operating on small chunks, there is never the > same kind of trade-off for memory use. A merkel tree or similar > could be calculated at a very detailed level potentially. Although > the precision from the perspective of reading from disk would likely > not matter much if we are in page cache anyway, very high precision > could be *very* useful when doing anti-entropy across data centers > on slow links. > There are devils in details, like how to select an appropriate ring > segment given that you don't have knowledge of the data density on > other nodes. But I feel that the overall idea/process seems very > promising. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-14064) Allow using file based certificates instead of keystores
[ https://issues.apache.org/jira/browse/CASSANDRA-14064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047092#comment-17047092 ] Benjamin Coverston commented on CASSANDRA-14064: Before I came looking in the project I actually created a similar patch. Is this something that the project is still interested in? Removing the dependency on JKS opens a path to integration with ACME and a bunch of other possibilities. > Allow using file based certificates instead of keystores > > > Key: CASSANDRA-14064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14064 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Streaming and Messaging >Reporter: Stefan Podkowinski >Priority: Normal > Labels: security > Fix For: 4.x > > Attachments: 0001-Add-keystore-less-ssl-handling.patch > > > The requirement of having to use a secure archive (keystore) for your > certificates and keys is not very common outside the Java ecosystem. Most > servers will accept individual certificate and key files and will come with > instructions how to generate those using openssl. This should also be an > option for Cassandra for users who see no reason in additionally having to > deal with keystores. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-4168) "Setup" section of tools/stress/README.txt needs update
[ https://issues.apache.org/jira/browse/CASSANDRA-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265600#comment-13265600 ] Benjamin Coverston commented on CASSANDRA-4168: --- +1 ant jar should be the default target. > "Setup" section of tools/stress/README.txt needs update > --- > > Key: CASSANDRA-4168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4168 > Project: Cassandra > Issue Type: Bug > Components: Tools >Affects Versions: 1.1.1 >Reporter: Tyler Patterson >Assignee: Brandon Williams >Priority: Minor > Labels: stress > > The README.txt file states "Run `ant` from the Cassandra source directory, > then Run `ant` from the contrib/stress directory." > The file needs to reflect the changes in the way stress is built. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4286) Enhance "Write Survey" mode to use a custom range
Benjamin Coverston created CASSANDRA-4286: - Summary: Enhance "Write Survey" mode to use a custom range Key: CASSANDRA-4286 URL: https://issues.apache.org/jira/browse/CASSANDRA-4286 Project: Cassandra Issue Type: New Feature Reporter: Benjamin Coverston Write survey mode could be more useful for sizing and tuning if you could define a custom range to survey. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3983) Change order of directory searching for c*.in.sh
[ https://issues.apache.org/jira/browse/CASSANDRA-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-3983: -- Attachment: 3983.txt Attaching a new set of scripts that all use the same resolution strategy for the include file starting first with the local directory, then the configured $HOME finally using the remaining directories where the packages may live. This patch does not address any issues with current packaging setting, or not setting the proper environment variables. > Change order of directory searching for c*.in.sh > > > Key: CASSANDRA-3983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3983 > Project: Cassandra > Issue Type: Improvement >Reporter: Nick Bailey >Assignee: paul cannon >Priority: Minor > Attachments: 3983.txt > > > When you have a c* package installed but attempt to run from a source build, > 'bin/cassandra' will search the packaged dirs for 'cassandra.in.sh' before > searching the dirs in your source build. We should reverse the order of that > search so it checks locally first. Also the init scripts for a package should > set the environment variables correctly so no searching needs to be done and > there is no worry of the init scripts loading the wrong file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4341) Small SSTable Segments Can Hurt Leveling Process
Benjamin Coverston created CASSANDRA-4341: - Summary: Small SSTable Segments Can Hurt Leveling Process Key: CASSANDRA-4341 URL: https://issues.apache.org/jira/browse/CASSANDRA-4341 Project: Cassandra Issue Type: Bug Reporter: Benjamin Coverston This concerns: static int MAX_COMPACTING_L0 = 32; Repair can create very small SSTable segments. We should consider moving to a threshold that takes into account the size of the files brought into compaction rather than the number of files for this and similar situations. Bringing the small files from L0 to L1 magnifies the issue. If there are too many very small files in L0 perhaps even an intermediate compaction would even reduce the magnifying effect of a L0 to L1 compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-4345) New (bootstrapping) Supplant Healthy Nodes
Benjamin Coverston created CASSANDRA-4345: - Summary: New (bootstrapping) Supplant Healthy Nodes Key: CASSANDRA-4345 URL: https://issues.apache.org/jira/browse/CASSANDRA-4345 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0.9 Reporter: Benjamin Coverston Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing: java.lang.NullPointerException ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice java.lang.NullPointerException ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice java.lang.NullPointerException ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice java.lang.NullPointerException --- Resulting in: INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88 .48 is the new owner WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48 INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops) INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops) INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes) ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice java.lang.NullPointerException at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104) at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122) at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93) at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564) at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128) at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283) at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365) at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326) at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631) at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2470) Secondary Indexes Build Very Slowly
Secondary Indexes Build Very Slowly --- Key: CASSANDRA-2470 URL: https://issues.apache.org/jira/browse/CASSANDRA-2470 Project: Cassandra Issue Type: Improvement Reporter: Benjamin Coverston While running repair I noticed that the time it took to run was easily dominated by building the secondary indexes. They currently build at a rate of < 200KB/Second. This means that indexing a 500MB file takes nearly an hour to index the file. Because this happens on the compaction thread it also causes repair to back up in general, which for very active systems is a very bad thing. I suggest we look at it in the hope that we can improve the rate in which we build the indexes by an order of magnitude. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2346) Windows: SSTableWriterTest Fails
[ https://issues.apache.org/jira/browse/CASSANDRA-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032195#comment-13032195 ] Benjamin Coverston commented on CASSANDRA-2346: --- Ran this with the patch and I'm still getting an failure for this test [1]. I'm also seeing a number of other failures in my tests, so I'm not sure if it's an environment issue or what. I'm going to tool around with it a bit more to see if it's java that's holding on to the file handles or something else. https://issues.apache.org/jira/browse/CASSANDRA-2346 > Windows: SSTableWriterTest Fails > > > Key: CASSANDRA-2346 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2346 > Project: Cassandra > Issue Type: Bug > Components: Tests >Affects Versions: 0.7.0 > Environment: Windows >Reporter: Benjamin Coverston >Assignee: Jonathan Ellis >Priority: Minor > Labels: windows > Fix For: 0.7.6 > > Attachments: 2346.txt > > > Fails with the following exception[1] > Which looks like there's a stream keeping this file open somewhere. > I looked for the culprit, but I was unable to identify it. Following it > through it appears that if the following two lines of code are executed from > IndexWriter.afterAppend: > ByteBufferUtil.writeWithShortLength(key.key, indexFile); > indexFile.writeLong(dataPosition); > Then the indexFile gets touched and that generally triggers this situation. > One thing interesting here is that an index file without any rows will > trigger a divide by zero exception [2]. > I looked at this quite a bit without being able to make much progress. > [1] > java.io.IOException: Failed to delete > C:\Users\Ben\AppData\Local\Temp\Keyspace12505092819180437091Indexed1\Keyspace1\Indexed1-f-0-Index.db > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:51) > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:41) > at > org.apache.cassandra.io.sstable.SSTableWriterTest.testRecoverAndOpen(SSTableWriterTest.java:79) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41) > at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > at org.junit.runners.ParentRunner.run(ParentRunner.java:220) > at org.junit.runner.JUnitCore.run(JUnitCore.java:159) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:97) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:196) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115) > [2] > ERROR 16:57:48,119 Error in ThreadPoolExecutor > java.lang.ArithmeticException: / by zero > at > org.apache.cassandra.io.sstable.SSTable.estimateRowsFromData(SSTable.java:218) > at > org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:291) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:942) > at > org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:935) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.Futu
[jira] [Commented] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033870#comment-13033870 ] Benjamin Coverston commented on CASSANDRA-1608: --- I like the LevelDB approach in almost every way except that recovery becomes more complicated than our current scenario. Right now we can basically ship files from one server to another without any special consideration, the reliance on a manifest requires us to solve a new problem in the case where a manifest is lost, or we want to recover from a snapshot. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040541#comment-13040541 ] Benjamin Coverston commented on CASSANDRA-1608: --- There's probably nothing that prevents us from doing that. Is our goal here to replace compaction entirely? The manifest information consists, minimally of the level information and ranges. For us ranges are easy as they are readily available when the SSTables are read in at restart, flushing, or compaction. Taking a stab at this I made the compaction manager abstract, then created a concrete implementation for the current compaction implementation. Happily hacking on a level based compaction manager I kept running into a delemma: Where do I store the level information. There are a few options: 1. The descriptor A hack, simple, but also adds information that probably wouldn't be used by any other compaction manager, yet it would be there. Unless we're moving head-long into a level-db approach I'm not super excited about this. 2. Store it on a per-sstable basis -in- the sstable: To continue along this path I would like to have a standard place to put "extra" metadata in the sstables. A header of some sort. I like the idea of using a metadata block in the SSTables to store this type of information. 3. Use an on-disk manifest. -- Pro: only my compaction manager needs to deal with this information, but there is a non-trivial amount of bookeeping that would need to be done to ensure this is kept up to day and valid. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040541#comment-13040541 ] Benjamin Coverston edited comment on CASSANDRA-1608 at 5/28/11 5:11 PM: There's probably nothing that prevents us from doing that. Is our goal here to replace compaction entirely? The manifest information consists, minimally of the level information and ranges. For us ranges are easy as they are readily available when the SSTables are read in at restart, flushing, or compaction. Taking a stab at this I made the compaction manager abstract, then created a concrete implementation for the current compaction implementation. Happily hacking on a level based compaction manager I kept running into a delemma: Where do I store the level information. There are a few options: 1. The descriptor A hack, simple, but also adds information that probably wouldn't be used by any other compaction manager, yet it would be there. Unless we're moving head-long into a level-db approach I'm not super excited about this. 2. Store it on a per-sstable basis -in- the sstable: To continue along this path I would like to have a standard place to put "extra" metadata in the sstables. A header of some sort. I like the idea of using a metadata block in the SSTables to store this type of information. 3. Use an on-disk manifest. -- Pro: only my compaction manager needs to deal with this information, but there is a non-trivial amount of bookeeping that would need to be done to ensure this is kept up to day and valid. EDIT: 4. This is probably the best option, create a new component type: METADATA_STORE which will hold namespaced key/value pairs on a per-sstable basis. was (Author: bcoverston): There's probably nothing that prevents us from doing that. Is our goal here to replace compaction entirely? The manifest information consists, minimally of the level information and ranges. For us ranges are easy as they are readily available when the SSTables are read in at restart, flushing, or compaction. Taking a stab at this I made the compaction manager abstract, then created a concrete implementation for the current compaction implementation. Happily hacking on a level based compaction manager I kept running into a delemma: Where do I store the level information. There are a few options: 1. The descriptor A hack, simple, but also adds information that probably wouldn't be used by any other compaction manager, yet it would be there. Unless we're moving head-long into a level-db approach I'm not super excited about this. 2. Store it on a per-sstable basis -in- the sstable: To continue along this path I would like to have a standard place to put "extra" metadata in the sstables. A header of some sort. I like the idea of using a metadata block in the SSTables to store this type of information. 3. Use an on-disk manifest. -- Pro: only my compaction manager needs to deal with this information, but there is a non-trivial amount of bookeeping that would need to be done to ensure this is kept up to day and valid. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1610) Pluggable Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043890#comment-13043890 ] Benjamin Coverston commented on CASSANDRA-1610: --- These changes are good. There is an added import to the StreamingTransferTest that doesn't need to be there: import org.apache.cassandra.db.compaction.CompactionManager; Overall the changes are good, I'm concerned that a few of the tests seem to fail with this patch when they don't fail in trunk right now. Get the fixes in and also add some documentation for the new switches in the cassandra.yaml then a +1. > Pluggable Compaction > > > Key: CASSANDRA-1610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1610 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Alan Liang >Priority: Minor > Labels: compaction > Fix For: 1.0 > > Attachments: 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0002-Pluggable-Compaction-and-Expiration.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch > > > In CASSANDRA-1608, I proposed some changes on how compaction works. I think > it also makes sense to allow the ability to have pluggable compaction per CF. > There could be many types of workloads where this makes sense. One example we > had at Digg was to completely throw away certain SSTables after N days. > This ticket addresses making compaction pluggable only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1610) Pluggable Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044357#comment-13044357 ] Benjamin Coverston commented on CASSANDRA-1610: --- Still looking at it, I'm still seeing failures on the org.apache.cassandra.db.KeyCacheTest and org.apache.cassandra.db.compaction.CompactionsPurgeTest. If you'er sure you're not seeing failures I'll push these changes out to another computer and test them there. Ben > Pluggable Compaction > > > Key: CASSANDRA-1610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1610 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Alan Liang >Priority: Minor > Labels: compaction > Fix For: 1.0 > > Attachments: 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0002-Pluggable-Compaction-and-Expiration.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch, > 0002-pluggable-compaction.patch > > > In CASSANDRA-1608, I proposed some changes on how compaction works. I think > it also makes sense to allow the ability to have pluggable compaction per CF. > There could be many types of workloads where this makes sense. One example we > had at Digg was to completely throw away certain SSTables after N days. > This ticket addresses making compaction pluggable only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1610) Pluggable Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044432#comment-13044432 ] Benjamin Coverston commented on CASSANDRA-1610: --- CompactionManager.java has a number of duplicate imports you probably want to clean up. Other than that this looks good. +1 > Pluggable Compaction > > > Key: CASSANDRA-1610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1610 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Alan Liang >Priority: Minor > Labels: compaction > Fix For: 1.0 > > Attachments: 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0002-Pluggable-Compaction-and-Expiration.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch > > > In CASSANDRA-1608, I proposed some changes on how compaction works. I think > it also makes sense to allow the ability to have pluggable compaction per CF. > There could be many types of workloads where this makes sense. One example we > had at Digg was to completely throw away certain SSTables after N days. > This ticket addresses making compaction pluggable only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1610) Pluggable Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1610: -- Attachment: 0002-pluggable-compaction.patch Moved the AbstractCompactionStrategy to be an interface. > Pluggable Compaction > > > Key: CASSANDRA-1610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1610 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Alan Liang >Priority: Minor > Labels: compaction > Fix For: 1.0 > > Attachments: 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0002-Pluggable-Compaction-and-Expiration.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch > > > In CASSANDRA-1608, I proposed some changes on how compaction works. I think > it also makes sense to allow the ability to have pluggable compaction per CF. > There could be many types of workloads where this makes sense. One example we > had at Digg was to completely throw away certain SSTables after N days. > This ticket addresses making compaction pluggable only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1610) Pluggable Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1610: -- Attachment: 0002-pluggable-compaction.patch Removed extraneous .orig > Pluggable Compaction > > > Key: CASSANDRA-1610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1610 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Alan Liang >Priority: Minor > Labels: compaction > Fix For: 1.0 > > Attachments: 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0001-move-compaction-code-into-own-package.patch, > 0002-Pluggable-Compaction-and-Expiration.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch, > 0002-pluggable-compaction.patch, 0002-pluggable-compaction.patch, > 0002-pluggable-compaction.patch > > > In CASSANDRA-1608, I proposed some changes on how compaction works. I think > it also makes sense to allow the ability to have pluggable compaction per CF. > There could be many types of workloads where this makes sense. One example we > had at Digg was to completely throw away certain SSTables after N days. > This ticket addresses making compaction pluggable only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2765) DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting
DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting -- Key: CASSANDRA-2765 URL: https://issues.apache.org/jira/browse/CASSANDRA-2765 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.8.0 Reporter: Benjamin Coverston At some point if the list isn't cleaned up with this symptom compactions will stop until the server is restarted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2765) DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting
[ https://issues.apache.org/jira/browse/CASSANDRA-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-2765: -- Attachment: 0001-removed-code-adding-all-sstables-to-the-view-builder.patch Attached patch removes code that adds all sstables to the compacting set. > DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting > -- > > Key: CASSANDRA-2765 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2765 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston > Attachments: > 0001-removed-code-adding-all-sstables-to-the-view-builder.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > At some point if the list isn't cleaned up with this symptom compactions will > stop until the server is restarted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2765) DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting
[ https://issues.apache.org/jira/browse/CASSANDRA-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston reassigned CASSANDRA-2765: - Assignee: Benjamin Coverston > DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting > -- > > Key: CASSANDRA-2765 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2765 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston >Assignee: Benjamin Coverston > Attachments: > 0001-removed-code-adding-all-sstables-to-the-view-builder.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > At some point if the list isn't cleaned up with this symptom compactions will > stop until the server is restarted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2765) DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting
[ https://issues.apache.org/jira/browse/CASSANDRA-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048782#comment-13048782 ] Benjamin Coverston commented on CASSANDRA-2765: --- Good call, I missed that part :). > DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting > -- > > Key: CASSANDRA-2765 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2765 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston >Assignee: Benjamin Coverston > Attachments: > 0001-removed-code-adding-all-sstables-to-the-view-builder.patch, 2765-v2.txt > > Original Estimate: 1h > Remaining Estimate: 1h > > At some point if the list isn't cleaned up with this symptom compactions will > stop until the server is restarted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2765) DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting
[ https://issues.apache.org/jira/browse/CASSANDRA-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048789#comment-13048789 ] Benjamin Coverston commented on CASSANDRA-2765: --- +1 > DataTracker.View.MarkCompacting adds ALL sstables and marks them as compacting > -- > > Key: CASSANDRA-2765 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2765 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston >Assignee: Benjamin Coverston > Fix For: 0.8.1 > > Attachments: > 0001-removed-code-adding-all-sstables-to-the-view-builder.patch, 2765-v2.txt > > Original Estimate: 1h > Remaining Estimate: 1h > > At some point if the list isn't cleaned up with this symptom compactions will > stop until the server is restarted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: 0001-leveldb-style-compaction.patch Adding a patch for leveldb style compaction. I see this as a 'good start' and I'm looking for some further input. I'm not going to be able to work on this for the next week or so so I'm putting it here to start some discussion on this approach. This implementation requires no durable manifest. Ranges are created at SSTable creation (flush or compaction) or sstable index creation. Exponent used for levels is 10. Preliminary runs show that high write rates do make level 0 to level 1 promotions back up substantially, but when cleared promotions out of level one seem to be very fast. I found the best performance by removing the compaction throughput throttling and setting concurrent compactors to 1. The SSTable size in this implementation is determined by the flush size in mb setting. The recovery path reads the list of SSTables, groups them by non-overlapping ranges then places each range in its appropriate level. Finally credit is due to the leveldb team as this design was inspired by the leveldb implementation. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet > Attachments: 0001-leveldb-style-compaction.patch > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
Cannot Create Duplicate Compaction Marker - Key: CASSANDRA-2769 URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.0 Reporter: Benjamin Coverston Concurrent compaction can trigger the following exception: I'll attach a patch to fix this. java.io.IOError: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) at org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.io.IOException: Unable to create compaction marker at org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049614#comment-13049614 ] Benjamin Coverston commented on CASSANDRA-2769: --- Yes > Cannot Create Duplicate Compaction Marker > - > > Key: CASSANDRA-2769 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston >Assignee: Sylvain Lebresne > > Concurrent compaction can trigger the following exception when two threads > compact the same sstable. DataTracker attempts to prevent this but apparently > not successfully. > java.io.IOError: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) > at > org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) > at > org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) > at > org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) > ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049614#comment-13049614 ] Benjamin Coverston edited comment on CASSANDRA-2769 at 6/15/11 5:37 AM: Yes, in both. was (Author: bcoverston): Yes > Cannot Create Duplicate Compaction Marker > - > > Key: CASSANDRA-2769 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston >Assignee: Sylvain Lebresne > > Concurrent compaction can trigger the following exception when two threads > compact the same sstable. DataTracker attempts to prevent this but apparently > not successfully. > java.io.IOError: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) > at > org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) > at > org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) > at > org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) > ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050890#comment-13050890 ] Benjamin Coverston commented on CASSANDRA-2769: --- I think Alan has a good point. I don't think it's an appropriate role of the data tracker to modify the set of sstables to be compacted in a task. It's been a bit of a struggle to hack around the behavior of the DataTracker to ensure my compactions happen in the order I want with the SSTables that I want. That should probably be the exclusive domain compaction strategy. > Cannot Create Duplicate Compaction Marker > - > > Key: CASSANDRA-2769 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston >Assignee: Sylvain Lebresne > Fix For: 0.8.2 > > Attachments: > 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch, > 0001-Do-compact-only-smallerSSTables.patch, > 0002-Only-compact-what-has-been-succesfully-marked-as-com.patch > > > Concurrent compaction can trigger the following exception when two threads > compact the same sstable. DataTracker attempts to prevent this but apparently > not successfully. > java.io.IOError: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) > at > org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) > at > org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) > at > org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) > ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1608) Redesigned Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050908#comment-13050908
]
Benjamin Coverston commented on CASSANDRA-1608:
---
The LDBCompaction task was changed to limit the size of the SSTables that are
output by the compaction itself. Once the size of rows compacted exceeds the
size of the default size in MB then it creates a new SSTable:
>>
if(position > cfs.metadata.getMemtableThroughputInMb() * 1024 * 1024
|| nni.hasNext() == false)
{
<<
It feels like a bit of a hack because an optimal flush size may not always be
an optimal storage size, but my goal was to try to keep the SSTable size in a
reasonably small range to make compactions into into level 1 fast.
I'll make some more modifications to the manifest s.t. there is a single path
for getting new SSTables (flushed and streamed) into the manifest. I found a
bug on the plane today where they were getting added to the manifest, but they
weren't being added to the queue that I was adding flushed SSTables to. I'll
get that into my next revision.
>>
In promote, do we need to check for all the removed ones being on the same
level? I can't think of a scenario where we're not merging from multiple
levels. If so I'd change that to an assert. (In fact there should be exactly
two levels involved, right?)
<<
I considered this. There are some boundary cases where every SSTable that gets
compacted will be in the same level. Most of them have to do with L+1 being
empty. Also sending the SSTables through the same compaction path will evict
expired tombstones before they end up in the next level where compactions
become increasingly unlikely.
>>
Did some surgery on getCompactionCandidates. Generally renamed things to be
more succinct. Feels like we getCompactionCandidates should do lower levels
before doing higher levels?
<<
Let's just say my naming conventions have been shaped by different influences
:) I wouldn't object to any of the new names you chose however.
RE: the order, it does feel like we should do lower levels before higher
levels, however one thing that we have to do is make sure that level-1 stays at
10 SSTables. The algorithm dictates that all of the level-0 candidates get
compacted with all of the candidates at level-1. This means that you need to
promote out of level-1 so that it is ~10 SSTables before you schedule a
compaction for level-0 promotion. Right now tuning this so that it is
performant is the biggest hurdle, I have made some improvements by watching the
CompactionExecutor, but I have a feeling that making this work is going to
require some subtle manipulation of the way that the CompactionExecutor handles
tasks.
> Redesigned Compaction
> -
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Chris Goffinet
> Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more
> thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the
> moment, compaction is kicked off based on a write access pattern, not read
> access pattern. In most cases, you want the opposite. You want to be able to
> track how well each SSTable is performing in the system. If we were to keep
> statistics in-memory of each SSTable, prioritize them based on most accessed,
> and bloom filter hit/miss ratios, we could intelligently group sstables that
> are being read most often and schedule them for compaction. We could also
> schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives
> us the ability to better utilize our bloom filters in a predictable manner.
> At the moment after a certain size, the bloom filters become less reliable.
> This would also allow us to group data most accessed. Currently the size of
> an SSTable can grow to a point where large portions of the data might not
> actually be accessed as often.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2769) Cannot Create Duplicate Compaction Marker
[ https://issues.apache.org/jira/browse/CASSANDRA-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051086#comment-13051086 ] Benjamin Coverston commented on CASSANDRA-2769: --- I'm sorry I didn't mean to imply it should be fixed _here_. I'll find a more appropriate venue to vent these frustrations :) > Cannot Create Duplicate Compaction Marker > - > > Key: CASSANDRA-2769 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2769 > Project: Cassandra > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Benjamin Coverston >Assignee: Sylvain Lebresne > Fix For: 0.8.2 > > Attachments: > 0001-0.8.0-Remove-useless-unmarkCompacting-in-doCleanup.patch, > 0001-Do-compact-only-smallerSSTables-v2.patch, > 0001-Do-compact-only-smallerSSTables.patch, > 0002-Only-compact-what-has-been-succesfully-marked-as-com-v2.patch, > 0002-Only-compact-what-has-been-succesfully-marked-as-com.patch > > > Concurrent compaction can trigger the following exception when two threads > compact the same sstable. DataTracker attempts to prevent this but apparently > not successfully. > java.io.IOError: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638) > at > org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:321) > at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:294) > at > org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:255) > at > org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:932) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:119) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:102) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:680) > Caused by: java.io.IOException: Unable to create compaction marker > at > org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:634) > ... 12 more -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Coverston updated CASSANDRA-1608:
--
Attachment: 1608-v3.txt
Attaching the latest version.
Levels are scored according to size rather than number of SSTables.
Performance kind of sucked until I made a few tweaks.
You can modify the LevelDB SSTable size by the following command (example) in
the CLI:
update column family Standard1 with compaction_strategy_options=[{
sstable_size_in_mb : 10 }];
Using a flush size of 64MB and an sstable_size_in_mb of 5 worked pretty well
for keeping compactions moving through the levels and handling new SSTables as
they entered the system.
I also enabled concurrent compactions which, to my surprise, helped
considerably. In testing I also removed compaction throttling, but in the end I
don't think it mattered too much for me.
This version also adds a manifest and recovery code to the mix. While running
you can cat the manifest, it's human readable, and quite beautiful to see the
levels interact with each other as SSTables are flushed and compactions roll
through the levels.
Right now I'm getting an exception on startup from the keycache, I'm going to
investigate that, but I think it may have to do with the fact that I am not
initializing the compaction manager _after_ the CFS.
> Redesigned Compaction
> -
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Chris Goffinet
> Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt,
> 1608-v3.txt
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more
> thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the
> moment, compaction is kicked off based on a write access pattern, not read
> access pattern. In most cases, you want the opposite. You want to be able to
> track how well each SSTable is performing in the system. If we were to keep
> statistics in-memory of each SSTable, prioritize them based on most accessed,
> and bloom filter hit/miss ratios, we could intelligently group sstables that
> are being read most often and schedule them for compaction. We could also
> schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives
> us the ability to better utilize our bloom filters in a predictable manner.
> At the moment after a certain size, the bloom filters become less reliable.
> This would also allow us to group data most accessed. Currently the size of
> an SSTable can grow to a point where large portions of the data might not
> actually be accessed as often.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1608) Redesigned Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052212#comment-13052212
]
Benjamin Coverston edited comment on CASSANDRA-1608 at 6/20/11 11:30 PM:
-
Attaching the latest version.
Levels are scored according to size rather than number of SSTables.
Performance kind of sucked until I made a few tweaks.
You can modify the LevelDB SSTable size by the following command (example) in
the CLI:
update column family Standard1 with compaction_strategy_options=[{
sstable_size_in_mb : 10 }];
Using a flush size of 64MB and an sstable_size_in_mb of 5 worked pretty well
for keeping compactions moving through the levels and handling new SSTables as
they entered the system.
I also enabled concurrent compactions which, to my surprise, helped
considerably. In testing I also removed compaction throttling, but in the end I
don't think it mattered too much for me.
This version also adds a manifest and recovery code to the mix. While running
you can cat the manifest, it's human readable, and quite beautiful to see the
levels interact with each other as SSTables are flushed and compactions roll
through the levels.
Right now I'm getting an exception on startup from the keycache, I'm going to
investigate that, but I think it may have to do with the fact that I am
initializing the compaction manager _after_ the CFS.
was (Author: bcoverston):
Attaching the latest version.
Levels are scored according to size rather than number of SSTables.
Performance kind of sucked until I made a few tweaks.
You can modify the LevelDB SSTable size by the following command (example) in
the CLI:
update column family Standard1 with compaction_strategy_options=[{
sstable_size_in_mb : 10 }];
Using a flush size of 64MB and an sstable_size_in_mb of 5 worked pretty well
for keeping compactions moving through the levels and handling new SSTables as
they entered the system.
I also enabled concurrent compactions which, to my surprise, helped
considerably. In testing I also removed compaction throttling, but in the end I
don't think it mattered too much for me.
This version also adds a manifest and recovery code to the mix. While running
you can cat the manifest, it's human readable, and quite beautiful to see the
levels interact with each other as SSTables are flushed and compactions roll
through the levels.
Right now I'm getting an exception on startup from the keycache, I'm going to
investigate that, but I think it may have to do with the fact that I am not
initializing the compaction manager _after_ the CFS.
> Redesigned Compaction
> -
>
> Key: CASSANDRA-1608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1608
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Chris Goffinet
> Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt,
> 1608-v3.txt
>
>
> After seeing the I/O issues in CASSANDRA-1470, I've been doing some more
> thinking on this subject that I wanted to lay out.
> I propose we redo the concept of how compaction works in Cassandra. At the
> moment, compaction is kicked off based on a write access pattern, not read
> access pattern. In most cases, you want the opposite. You want to be able to
> track how well each SSTable is performing in the system. If we were to keep
> statistics in-memory of each SSTable, prioritize them based on most accessed,
> and bloom filter hit/miss ratios, we could intelligently group sstables that
> are being read most often and schedule them for compaction. We could also
> schedule lower priority maintenance on SSTable's not often accessed.
> I also propose we limit the size of each SSTable to a fix sized, that gives
> us the ability to better utilize our bloom filters in a predictable manner.
> At the moment after a certain size, the bloom filters become less reliable.
> This would also allow us to group data most accessed. Currently the size of
> an SSTable can grow to a point where large portions of the data might not
> actually be accessed as often.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: 1608-v4.txt Fixed exception on startup. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: 1608-v5.txt Added range filters to reads. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: 2608-v6.txt Added a patch with fixed range filters. With ~1100 sstables average latency is substantially increased (~5-10x). I'm pretty sure that in order to improve on this well need to implement an interval tree to get a non-linear search time for overlapping sstables in interval queries. The problem here is that there aren't any really good RBtree or even binary tree implementations that I have found in the dependencies that we currently have, and I really don't want to muddy this ticket up with that effort. There are some potentially useful structures in UIMA that I can use to base the implementation of an interval tree off of, but right now I'm leaning toward doing this in a separate ticket. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt, 2608-v6.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: (was: 2608-v6.txt) > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054592#comment-13054592 ] Benjamin Coverston commented on CASSANDRA-1608: --- It won't because you can't do a simple binary search for a range, it's really a problem of intersection rather than matching, and comparators alone don't solve problem of: give me all the intersecting ranges for this set without having to compare every range for intersection. Nearly every interval intersection algorithm depends on tree traversal, and while many of the existing collections are based on binary, or red-black trees they don't expose the methods necessary for traversal, only the comparator is exposed used to build the tree and the models only expose either "iterate over everything" or "search for the thing I want". > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054628#comment-13054628 ] Benjamin Coverston commented on CASSANDRA-1608: --- You're right, I'm thick. I just had to go down to the node itself, the Entry object gives me the interface I need. -- Ben Coverston Director of Operations DataStax -- The Apache Cassandra Company http://www.datastax.com/ > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054628#comment-13054628 ] Benjamin Coverston edited comment on CASSANDRA-1608 at 6/24/11 7:26 PM: You're right, I'm thick. I just had to go down to the node itself, the Entry object gives me the interface I need. was (Author: ben.covers...@datastax.com): You're right, I'm thick. I just had to go down to the node itself, the Entry object gives me the interface I need. -- Ben Coverston Director of Operations DataStax -- The Apache Cassandra Company http://www.datastax.com/ > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: 1608-v7.txt Added an interval tree to cull sstables that are not needed for point and range queries. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt, 1608-v7.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: 1608-v8.txt First the good: 1. Modified the code s.t. tombstone purge during minor compactions use the interval tree to prune the list of SSTables speeding up compactions by at least an order of magnitude where the number of SSTables in a column family exceeds ~500. 2. Tested reads and writes. Write speeds (unsurprisingly) are not affected by this compaction strategy. Reads seem to keep up as well. The interval tree does a good job here making sure that bloom filters are only queried only for those SSTables that fall into the queried range. 3. Three successive runs of stress inserting 10M keys resulted in ~3GB of data stored in leveldb. By comparison, the same run using the tiered (default) strategy resulted in ~8GB of data. The Meh: Compactions do back up when setting the flush size to 64MB and the leveled SSTable size to anywhere between 5-10MB. On the upside, if your load has peaks and quieter times this compaction strategy will trigger a periodic check to "catch up" if all event-scheduled compactions complete. Interestingly this extra IO has an upside. For datasets that frequently overwrite old data that has already been flushed to disk there is the potential for substantial de-duplication of data. Further, during reads the number of rows that would need to be merged for a single row is bound by the number of levels + the number of un-leveled sstables. > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 0001-leveldb-style-compaction.patch, 1608-v2.txt, > 1608-v3.txt, 1608-v4.txt, 1608-v5.txt, 1608-v7.txt, 1608-v8.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-47) SSTable compression
[ https://issues.apache.org/jira/browse/CASSANDRA-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062244#comment-13062244 ] Benjamin Coverston commented on CASSANDRA-47: - I was worried a bit about this ticket while working on 1608 because it had the potential to be a hairy merge. Looking through the patch however the implementation looks clean, unobtrusive and well encapsulated. This makes me really excited. Contrary to my initial fears I'm anticipating that this will actually help with 1608 and probably even help increase IO throughput in the data directory in a dramatic way. +1 for this approach. > SSTable compression > --- > > Key: CASSANDRA-47 > URL: https://issues.apache.org/jira/browse/CASSANDRA-47 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Labels: compression > Fix For: 1.0 > > Attachments: CASSANDRA-47.patch, snappy-java-1.0.3-rc4.jar > > > We should be able to do SSTable compression which would trade CPU for I/O > (almost always a good trade). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: (was: 1608-v4.txt) > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 1608-v2.txt, 1608-v8.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-1608) Redesigned Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Coverston updated CASSANDRA-1608: -- Attachment: (was: 0001-leveldb-style-compaction.patch) > Redesigned Compaction > - > > Key: CASSANDRA-1608 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1608 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Chris Goffinet >Assignee: Benjamin Coverston > Attachments: 1608-v2.txt, 1608-v8.txt > > > After seeing the I/O issues in CASSANDRA-1470, I've been doing some more > thinking on this subject that I wanted to lay out. > I propose we redo the concept of how compaction works in Cassandra. At the > moment, compaction is kicked off based on a write access pattern, not read > access pattern. In most cases, you want the opposite. You want to be able to > track how well each SSTable is performing in the system. If we were to keep > statistics in-memory of each SSTable, prioritize them based on most accessed, > and bloom filter hit/miss ratios, we could intelligently group sstables that > are being read most often and schedule them for compaction. We could also > schedule lower priority maintenance on SSTable's not often accessed. > I also propose we limit the size of each SSTable to a fix sized, that gives > us the ability to better utilize our bloom filters in a predictable manner. > At the moment after a certain size, the bloom filters become less reliable. > This would also allow us to group data most accessed. Currently the size of > an SSTable can grow to a point where large portions of the data might not > actually be accessed as often. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
