[Cassandra Wiki] Trivial Update of "Operations_JP" by MakiWatanabe
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Cassandra Wiki" for change notification. The "Operations_JP" page has been changed by MakiWatanabe. The comment on this change is: Fix typo. http://wiki.apache.org/cassandra/Operations_JP?action=diff&rev1=103&rev2=104 -- 3. もう一つのオプションは、単純に'nodetool repair'を全ノードで実施した後、compactionを実行してtombstoneをexpireするすることです。以降はread repairと通常の'nodetool repair'によってシステムの整合性が回復します。この方法は前の手法よりも実施が容易ですが、より多くの「削除の喪失」が発生することになるでしょう。 === ノード障害への対処 === - ノードが一時的な停止の後で回復した場合にデータの整合性を回復するには通常のrepair機構で十分でしょう。しかし注意して頂きたいのは、ノードの停止中に更新が実行され、設定されたGCGraceSeconds（標準値は10日）以内にノードのrepairが行われなかった場合は、その期間の削除操作が完全に失われるということです。あなたのアプリ-ションが削除をまったく行わないのでない限り、このようなケースでは障害ノードのデータを完全に削除し、再ブートストラップし、従来使用していたトークンについてremovetokenを実行する必要があります（下記を参照）。 + ノードが一時的な停止の後で回復した場合にデータの整合性を回復するには通常のrepair機構で十分でしょう。しかし注意して頂きたいのは、ノードの停止中に更新が実行され、設定されたGCGraceSeconds（標準値は10日）以内にノードのrepairが行われなかった場合は、その期間の削除操作が完全に失われるということです。あなたのアプリケーションが削除をまったく行わないのでない限り、このようなケースでは障害ノードのデータを完全に削除し、再ブートストラップし、従来使用していたトークンについてremovetokenを実行する必要があります（下記を参照）。 ノードが完全に停止し、回復の見込みがない場合は、2つの選択肢があります:
[jira] [Issue Comment Edited] (CASSANDRA-2371) Removed/Dead Node keeps reappearing
[ https://issues.apache.org/jira/browse/CASSANDRA-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014389#comment-13014389 ] Jonathan Ellis edited comment on CASSANDRA-2371 at 4/1/11 4:19 AM: --- Did you get a chance to try a patched build? was (Author: jbellis): Did you git a chance to try a patched build? > Removed/Dead Node keeps reappearing > --- > > Key: CASSANDRA-2371 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2371 > Project: Cassandra > Issue Type: Bug > Components: Tools >Affects Versions: 0.7.3, 0.7.4 > Environment: Large Amazon EC2 instances. Ubuntu 10.04.2 >Reporter: techlabs >Assignee: Brandon Williams >Priority: Minor > Fix For: 0.7.5 > > Attachments: 2371.txt > > > The removetoken option does not seem to work. The original node 10.240.50.63 > comes back into the ring, even after the EC2 instance is no longer in > existence. Originally I tried to add a new node 10.214.103.224 with the same > token, but there were some complications with that. I have pasted below all > the INFO log entries found with greping the system log files. > Seems to be a similar issue seen with > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Ghost-node-showing-up-in-the-ring-td6198180.html > > INFO [GossipStage:1] 2011-03-16 00:54:31,590 StorageService.java (line 745) > Nodes /10.214.103.224 and /10.240.50.63 have the same token > 957044156965139000. /10.214.103.224 is the new owner > INFO [GossipStage:1] 2011-03-16 17:26:51,083 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:27:24,767 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:29:30,191 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:31:35,609 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:33:39,440 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-23 17:22:55,520 StorageService.java (line 865) > Removing token 957044156965139000 for /10.240.50.63 > INFO [GossipStage:1] 2011-03-10 03:52:37,299 Gossiper.java (line 608) Node > /10.240.50.63 is now part of the cluster > INFO [GossipStage:1] 2011-03-10 03:52:37,545 Gossiper.java (line 600) > InetAddress /10.240.50.63 is now UP > INFO [HintedHandoff:1] 2011-03-10 03:53:36,168 HintedHandOffManager.java > (line 304) Started hinted handoff for endpoint /10.240.50.63 > INFO [HintedHandoff:1] 2011-03-10 03:53:36,169 HintedHandOffManager.java > (line 360) Finished hinted handoff of 0 rows to endpoint /10.240.50.63 > INFO [GossipStage:1] 2011-03-15 23:23:43,770 Gossiper.java (line 623) Node > /10.240.50.63 has restarted, now UP again > INFO [GossipStage:1] 2011-03-15 23:23:43,771 StorageService.java (line 726) > Node /10.240.50.63 state jump to normal > INFO [HintedHandoff:1] 2011-03-15 23:28:48,957 HintedHandOffManager.java > (line 304) Started hinted handoff for endpoint /10.240.50.63 > INFO [HintedHandoff:1] 2011-03-15 23:28:48,958 HintedHandOffManager.java > (line 360) Finished hinted handoff of 0 rows to endpoint /10.240.50.63 > INFO [ScheduledTasks:1] 2011-03-15 23:37:25,071 Gossiper.java (line 226) > InetAddress /10.240.50.63 is now dead. > INFO [GossipStage:1] 2011-03-16 00:54:31,590 StorageService.java (line 745) > Nodes /10.214.103.224 and /10.240.50.63 have the same token > 957044156965139000. /10.214.103.224 is the new owner > WARN [GossipStage:1] 2011-03-16 00:54:31,590 TokenMetadata.java (line 115) > Token 957044156965139000 changing ownership from > /10.240.50.63 to /10.214.103.224 > INFO [GossipStage:1] 2011-03-18 23:37:09,158 Gossiper.java (line 610) Node > /10.240.50.63 is now part of the cluster > INFO [GossipStage:1] 2011-03-21 23:37:10,421 Gossiper.java (line 610) Node > /10.240.50.63 is now part of the cluster > INFO [GossipStage:1] 2011-03-21 23:37:10,421 StorageService.java (line 726) > Node /10.240.50.63 state jump to normal > INFO [GossipStage:1] 2011-03-23 17:22:55,520 StorageService.java (line 865) > Removing token 957044156965139000 for /10.240.50.63 > INFO [ScheduledTasks:1] 2011-03-23 17:22:55,521 HintedHandOffManager.java > (line 210) Deleting any stored hints for 10.240.50.63 -- This message is automatically generated by JIRA. For more information on JIR
[jira] [Commented] (CASSANDRA-2371) Removed/Dead Node keeps reappearing
[ https://issues.apache.org/jira/browse/CASSANDRA-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014389#comment-13014389 ] Jonathan Ellis commented on CASSANDRA-2371: --- Did you git a chance to try a patched build? > Removed/Dead Node keeps reappearing > --- > > Key: CASSANDRA-2371 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2371 > Project: Cassandra > Issue Type: Bug > Components: Tools >Affects Versions: 0.7.3, 0.7.4 > Environment: Large Amazon EC2 instances. Ubuntu 10.04.2 >Reporter: techlabs >Assignee: Brandon Williams >Priority: Minor > Fix For: 0.7.5 > > Attachments: 2371.txt > > > The removetoken option does not seem to work. The original node 10.240.50.63 > comes back into the ring, even after the EC2 instance is no longer in > existence. Originally I tried to add a new node 10.214.103.224 with the same > token, but there were some complications with that. I have pasted below all > the INFO log entries found with greping the system log files. > Seems to be a similar issue seen with > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Ghost-node-showing-up-in-the-ring-td6198180.html > > INFO [GossipStage:1] 2011-03-16 00:54:31,590 StorageService.java (line 745) > Nodes /10.214.103.224 and /10.240.50.63 have the same token > 957044156965139000. /10.214.103.224 is the new owner > INFO [GossipStage:1] 2011-03-16 17:26:51,083 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:27:24,767 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:29:30,191 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:31:35,609 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-19 17:33:39,440 StorageService.java (line 865) > Removing token 957044156965139000 for /10.214.103.224 > INFO [GossipStage:1] 2011-03-23 17:22:55,520 StorageService.java (line 865) > Removing token 957044156965139000 for /10.240.50.63 > INFO [GossipStage:1] 2011-03-10 03:52:37,299 Gossiper.java (line 608) Node > /10.240.50.63 is now part of the cluster > INFO [GossipStage:1] 2011-03-10 03:52:37,545 Gossiper.java (line 600) > InetAddress /10.240.50.63 is now UP > INFO [HintedHandoff:1] 2011-03-10 03:53:36,168 HintedHandOffManager.java > (line 304) Started hinted handoff for endpoint /10.240.50.63 > INFO [HintedHandoff:1] 2011-03-10 03:53:36,169 HintedHandOffManager.java > (line 360) Finished hinted handoff of 0 rows to endpoint /10.240.50.63 > INFO [GossipStage:1] 2011-03-15 23:23:43,770 Gossiper.java (line 623) Node > /10.240.50.63 has restarted, now UP again > INFO [GossipStage:1] 2011-03-15 23:23:43,771 StorageService.java (line 726) > Node /10.240.50.63 state jump to normal > INFO [HintedHandoff:1] 2011-03-15 23:28:48,957 HintedHandOffManager.java > (line 304) Started hinted handoff for endpoint /10.240.50.63 > INFO [HintedHandoff:1] 2011-03-15 23:28:48,958 HintedHandOffManager.java > (line 360) Finished hinted handoff of 0 rows to endpoint /10.240.50.63 > INFO [ScheduledTasks:1] 2011-03-15 23:37:25,071 Gossiper.java (line 226) > InetAddress /10.240.50.63 is now dead. > INFO [GossipStage:1] 2011-03-16 00:54:31,590 StorageService.java (line 745) > Nodes /10.214.103.224 and /10.240.50.63 have the same token > 957044156965139000. /10.214.103.224 is the new owner > WARN [GossipStage:1] 2011-03-16 00:54:31,590 TokenMetadata.java (line 115) > Token 957044156965139000 changing ownership from > /10.240.50.63 to /10.214.103.224 > INFO [GossipStage:1] 2011-03-18 23:37:09,158 Gossiper.java (line 610) Node > /10.240.50.63 is now part of the cluster > INFO [GossipStage:1] 2011-03-21 23:37:10,421 Gossiper.java (line 610) Node > /10.240.50.63 is now part of the cluster > INFO [GossipStage:1] 2011-03-21 23:37:10,421 StorageService.java (line 726) > Node /10.240.50.63 state jump to normal > INFO [GossipStage:1] 2011-03-23 17:22:55,520 StorageService.java (line 865) > Removing token 957044156965139000 for /10.240.50.63 > INFO [ScheduledTasks:1] 2011-03-23 17:22:55,521 HintedHandOffManager.java > (line 210) Deleting any stored hints for 10.240.50.63 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of "RunningCassandraInEclipse" by DavidWeiser
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Cassandra Wiki" for
change notification.
The "RunningCassandraInEclipse" page has been changed by DavidWeiser.
http://wiki.apache.org/cassandra/RunningCassandraInEclipse?action=diff&rev1=20&rev2=21
--
INFO 22:01:25,691 Binding thrift service to localhost/127.0.0.1:9160
}}}
+ '''Note''': You may find that you receive an error like:
+ {{{java.io.IOError: java.io.IOException: unable to mkdirs
/var/lib/cassandra/data}}}
+
+ when running org.apache.cassandra.thrift.CassandraDaemon. This is because
the cassandra.yaml file contains this path. Change these paths to a path which
you have access to.
+
Try running the cassandra-cli as per CassandraCli, and you should be able to
connect to localhost/9160.
Problems? contact me (rschildmeijer) in #cassandra on irc
([[http://freenode.net/irc_servers.shtml|freenode]]).
[Cassandra Wiki] Update of "RunningCassandraInEclipse" by DavidWeiser
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Cassandra Wiki" for
change notification.
The "RunningCassandraInEclipse" page has been changed by DavidWeiser.
The comment on this change is: Added clarification.
http://wiki.apache.org/cassandra/RunningCassandraInEclipse?action=diff&rev1=19&rev2=20
--
-ea -Xmx1G
}}}
- Below are another set of VM arguments for running Cassandra 0.7+ in Windows.
Notice that for config file property has changed to cassandra.config from
storage-config in 0.6 and that all the files need to have file: perpended in
Windows.
+ Below are another set of VM arguments for running Cassandra 0.7+ in Windows.
Notice that for config file property has changed to cassandra.config from
storage-config in 0.6 and that all the files need to have "file:" perpended,
for example:
{{{
-Dcassandra.config=file:C:\Users\Joaquin\workspace\cassandra-3\conf\cassandra.yaml
-Dcassandra-foreground
-ea -Xmx1G
-Dlog4j.configuration=file:C:\Users\Joaquin\workspace\cassandra-3\conf\log4j-server.properties
+ }}}
+
+ or
+ {{{
+
-Dcassandra.config=file:/home/david/programming/java/cassandra/conf/cassandra.yaml
+ -Dcassandra-foreground
+ -ea -Xmx1G
+
-Dlog4j.configuration=file:/home/david/programming/java/cassandra/conf/log4j-server.properties
}}}
Make sure to change the storage-config property so it defines the path to
your storage-conf.xml file.
[jira] [Commented] (CASSANDRA-2156) Compaction Throttling
[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014298#comment-13014298
]
Ryan King commented on CASSANDRA-2156:
--
This has been a big improvement for us in production. It'd be nice to get more
eyes on it for 0.8.
> Compaction Throttling
> -
>
> Key: CASSANDRA-2156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2156
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Stu Hood
> Fix For: 0.8
>
> Attachments:
> 0005-Throttle-total-compaction-to-a-configurable-throughput.txt,
> for-0.6-0001-Throttle-compaction-to-a-fixed-throughput.txt,
> for-0.6-0002-Make-compaction-throttling-configurable.txt
>
>
> Compaction is currently relatively bursty: we compact as fast as we can, and
> then we wait for the next compaction to be possible ("hurry up and wait").
> Instead, to properly amortize compaction, you'd like to compact exactly as
> fast as you need to to keep the sstable count under control.
> For every new level of compaction, you need to increase the rate that you
> compact at: a rule of thumb that we're testing on our clusters is to
> determine the maximum number of buckets a node can support (aka, if the 15th
> bucket holds 750 GB, we're not going to have more than 15 buckets), and then
> multiply the flush throughput by the number of buckets to get a minimum
> compaction throughput to maintain your sstable count.
> Full explanation: for a min compaction threshold of {{T}}, the bucket at
> level {{N}} can contain {{SsubN = T^N}} 'units' (unit == memtable's worth of
> data on disk). Every time a new unit is added, it has a {{1/SsubN}} chance of
> causing the bucket at level N to fill. If the bucket at level N fills, it
> causes {{SsubN}} units to be compacted. So, for each active level in your
> system you have {{SubN * 1 / SsubN}}, or {{1}} amortized unit to compact any
> time a new unit is added.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2387) Make it possible for pig to understand packed data
[ https://issues.apache.org/jira/browse/CASSANDRA-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-2387: Attachment: 2387-v4.txt Working version of v2/v3 and folding in the compose method that was added to AbstractType as part of CASSANDRA-2262 which makes CASSANDRA-2403 done as part of this. Still have to test writing which will be tonight probably. Also Brandon had some ideas about optimization. > Make it possible for pig to understand packed data > -- > > Key: CASSANDRA-2387 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2387 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna > Labels: contrib, hadoop, pig > Attachments: 2387-1.txt, 2387-v2.txt, 2387-v3.txt, 2387-v4.txt, > loadstorecaster-patch.txt > > > Packed values are throwing off pig. This ticket is to make it so pig can > interpret packed values. Originally we thought we could just use a > loadcaster. However, the only way we know how we can do it now is to get the > schema through thrift and essentially perform the function of the loadcaster > in the getNext method. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1087470 - /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
Author: jbellis
Date: Thu Mar 31 22:40:52 2011
New Revision: 1087470
URL: http://svn.apache.org/viewvc?rev=1087470&view=rev
Log:
add seek position, fileLength too
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java?rev=1087470&r1=1087469&r2=1087470&view=diff
==
---
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
(original)
+++
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
Thu Mar 31 22:40:52 2011
@@ -338,7 +338,8 @@ public class BufferedRandomAccessFile ex
throw new IllegalArgumentException("new position should not be
negative");
if (isReadOnly() && newPosition > fileLength)
-throw new EOFException("unable to seek past the end of " +
filePath + " in read-only mode.");
+throw new EOFException(String.format("unable to seek to position
%d in %s (%d bytes) in read-only mode",
+ newPosition, filePath,
fileLength));
current = newPosition;
svn commit: r1087469 - /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
Author: jbellis
Date: Thu Mar 31 22:37:04 2011
New Revision: 1087469
URL: http://svn.apache.org/viewvc?rev=1087469&view=rev
Log:
add file path to seek-past-eof exception
patch by jbellis
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java?rev=1087469&r1=1087468&r2=1087469&view=diff
==
---
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
(original)
+++
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/io/util/BufferedRandomAccessFile.java
Thu Mar 31 22:37:04 2011
@@ -338,7 +338,7 @@ public class BufferedRandomAccessFile ex
throw new IllegalArgumentException("new position should not be
negative");
if (isReadOnly() && newPosition > fileLength)
-throw new EOFException("unable to seek past the end of the file in
read-only mode.");
+throw new EOFException("unable to seek past the end of " +
filePath + " in read-only mode.");
current = newPosition;
[jira] [Created] (CASSANDRA-2411) log why a SSTable is being deleted
log why a SSTable is being deleted
--
Key: CASSANDRA-2411
URL: https://issues.apache.org/jira/browse/CASSANDRA-2411
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.7.4
Reporter: Robert Coli
Priority: Minor
./src/java/org/apache/cassandra/io/sstable/SSTable.java: 147
Has "logger.info("Deleted " + desc); ".
This combined with the JVM not being able to delete files until a GC has run
means that restarting a node usually prompts a flood of log messages like :
"Deleted /mnt/var/cassandra/data/Digg/UserActivity-10658-Data.db"
I believe that I am not the only operator who reads a log upon restart that
says "I deleted your data files" and becomes concerned.
Now, personally, I have read the code and understand the conditions under which
these files are deleted and so no longer get frightened. For new operators and
people who may feel less comfortable reading the code, specifying WHY the file
has been deleted ("Deleted because it was obsolete and marked for
deletion as a result of compaction.") seems likely to reduce blood pressure and
operator stress levels. :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2410) JDBC ResultSet does not honor column value typing for the CF and uses default validator for all column value types.
JDBC ResultSet does not honor column value typing for the CF and uses default
validator for all column value types.
---
Key: CASSANDRA-2410
URL: https://issues.apache.org/jira/browse/CASSANDRA-2410
Project: Cassandra
Issue Type: Bug
Components: API
Reporter: Rick Shaw
Priority: Minor
Fix For: 0.8
Assume a CF declared in CQL as :
{code}
CREATE COLUMNFAMILY TestCF(KEY utf8 PRIMARY KEY,description utf8, anumber int)
WITH comparator = ascii AND default_validation = long;
{code}
If the {{ResultSet}} is fetched thusly:
{code}
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery(query);
String description;
Integer anumber;
while (rs.next())
{
description = rs.getString(1);
System.out.print("description : "+ description);
anumber = rs.getInt(2);
System.out.print("anumber : "+ anumber);
}
{code}
It will immediately fail with a message of:
{code}
org.apache.cassandra.db.marshal.MarshalException: A long is exactly 8 bytes: 16
at org.apache.cassandra.db.marshal.LongType.getString(LongType.java:66)
at org.apache.cassandra.cql.jdbc.TypedColumn.(TypedColumn.java:45)
at
org.apache.cassandra.cql.jdbc.ColumnDecoder.makeCol(ColumnDecoder.java:158)
at
org.apache.cassandra.cql.jdbc.CassandraResultSet.next(CassandraResultSet.java:1073)
at da.access.testing.TestJDBC.selectAll(TestJDBC.java:83)
...
{code}
It appears that the {{makeCol}} method of {{ColumnDecoder.java}} chooses NOT to
use the {{CfDef}} to look up the possible occurrence of a column? That's not
right. Right?
{code}
/** constructs a typed column */
public TypedColumn makeCol(String keyspace, String columnFamily, byte[]
name, byte[] value)
{
CfDef cfDef = cfDefs.get(String.format("%s.%s", keyspace,
columnFamily));
AbstractType comparator = getComparator(keyspace, columnFamily,
Specifier.Comparator, cfDef);
AbstractType validator = getComparator(keyspace, columnFamily,
Specifier.Validator, null);
return new TypedColumn(comparator, name, validator, value);
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2358) CLI doesn't handle inserting negative integers
[
https://issues.apache.org/jira/browse/CASSANDRA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014172#comment-13014172
]
Hudson commented on CASSANDRA-2358:
---
Integrated in Cassandra #822 (See
[https://hudson.apache.org/hudson/job/Cassandra/822/])
add negative number support to cli, trunk version
patch by Pavel Yaskevich for CASSANDRA-2358
> CLI doesn't handle inserting negative integers
> --
>
> Key: CASSANDRA-2358
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2358
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Affects Versions: 0.7.0
>Reporter: Tyler Hobbs
>Assignee: Pavel Yaskevich
>Priority: Trivial
> Fix For: 0.7.5
>
> Attachments: CASSANDRA-2358-trunk.patch, CASSANDRA-2358.patch
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> The CLI raises a syntax error when trying to insert negative integers:
> {noformat}
> [default@Keyspace1] set StandardInteger['key'][-12] = 'val';
> Syntax error at position 28: mismatched character '1' expecting '-'
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.
[ https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stu Hood updated CASSANDRA-2388: Labels: hadoop inputformat (was: ) > ColumnFamilyRecordReader fails for a given split because a host is down, even > if records could reasonably be read from other replica. > - > > Key: CASSANDRA-2388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2388 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Reporter: Eldon Stegall > Labels: hadoop, inputformat > Attachments: 0002_On_TException_try_next_split.patch > > > ColumnFamilyRecordReader only tries the first location for a given split. We > should try multiple locations for a given split. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2409) Add INSERT support to CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014139#comment-13014139 ] Pavel Yaskevich commented on CASSANDRA-2409: Maybe I have a solution which will satisfy both sides - we can just make INSERT as an alias for UPDATE, so no need to implement anything just a simple change in the grammar, what do you think? > Add INSERT support to CQL > - > > Key: CASSANDRA-2409 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2409 > Project: Cassandra > Issue Type: Bug > Components: API >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 0.8 > > > There are two reasons to support INSERT: > - It helps new users feel comfortable (everyone's first statement will be to > try to INSERT something, we should make that a positive experience instead of > slapping them) > - Even though it is synonymous with update it is still useful in your code to > have both to help communicate intent, similar to choosing good variable names > The only downside is explaining how INSERT isn't a "true" insert because it > doesn't error out if the row already exists -- but we already have to explain > that same concept for UPDATE; the cognitive load is extremely minor. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2409) Add INSERT support to CQL
[
https://issues.apache.org/jira/browse/CASSANDRA-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014136#comment-13014136
]
Eric Evans commented on CASSANDRA-2409:
---
I don't think this is a good idea. {{UPDATE}} already has semantics that are
going to be a surprise to folks coming from SQL. Implementing {{INSERT}} to
have identical semantics is going to be (IMO) more surprising still, not less.
But, I don't want to get wrapped up in a protracted debate about it, so I'll
leave this as "-0", and let you guys carry-on as you see fit.
P.S. Please don't wait too long, the freeze is looming.
> Add INSERT support to CQL
> -
>
> Key: CASSANDRA-2409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2409
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Jonathan Ellis
>Assignee: Pavel Yaskevich
> Fix For: 0.8
>
>
> There are two reasons to support INSERT:
> - It helps new users feel comfortable (everyone's first statement will be to
> try to INSERT something, we should make that a positive experience instead of
> slapping them)
> - Even though it is synonymous with update it is still useful in your code to
> have both to help communicate intent, similar to choosing good variable names
> The only downside is explaining how INSERT isn't a "true" insert because it
> doesn't error out if the row already exists -- but we already have to explain
> that same concept for UPDATE; the cognitive load is extremely minor.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
svn commit: r1087404 - in /cassandra/trunk: src/java/org/apache/cassandra/cli/Cli.g test/unit/org/apache/cassandra/cli/CliTest.java
Author: jbellis
Date: Thu Mar 31 18:39:37 2011
New Revision: 1087404
URL: http://svn.apache.org/viewvc?rev=1087404&view=rev
Log:
add negative number support to cli, trunk version
patch by Pavel Yaskevich for CASSANDRA-2358
Modified:
cassandra/trunk/src/java/org/apache/cassandra/cli/Cli.g
cassandra/trunk/test/unit/org/apache/cassandra/cli/CliTest.java
Modified: cassandra/trunk/src/java/org/apache/cassandra/cli/Cli.g
URL:
http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/cli/Cli.g?rev=1087404&r1=1087403&r2=1087404&view=diff
==
--- cassandra/trunk/src/java/org/apache/cassandra/cli/Cli.g (original)
+++ cassandra/trunk/src/java/org/apache/cassandra/cli/Cli.g Thu Mar 31 18:39:37
2011
@@ -423,6 +423,7 @@ attrValueString
attrValueInt
: IntegerPositiveLiteral
+ | IntegerNegativeLiteral
;
attrValueDouble
@@ -471,11 +472,11 @@ columnFamily
;
rowKey
-: (Identifier | StringLiteral | IntegerPositiveLiteral | functionCall)
+: (Identifier | StringLiteral | IntegerPositiveLiteral |
IntegerNegativeLiteral | functionCall)
;
value
-: (Identifier | IntegerPositiveLiteral | StringLiteral | functionCall)
+: (Identifier | IntegerPositiveLiteral | IntegerNegativeLiteral |
StringLiteral | functionCall)
;
functionCall
@@ -484,7 +485,7 @@ functionCall
;
functionArgument
-: Identifier | StringLiteral | IntegerPositiveLiteral
+: Identifier | StringLiteral | IntegerPositiveLiteral |
IntegerNegativeLiteral
;
startKey
@@ -496,7 +497,7 @@ endKey
;
columnOrSuperColumn
- : (Identifier | IntegerPositiveLiteral | StringLiteral | functionCall)
+ : (Identifier | IntegerPositiveLiteral | IntegerNegativeLiteral |
StringLiteral | functionCall)
;
host
@@ -519,8 +520,8 @@ port
;
incrementValue
-: IntegerNegativeLiteral
-| IntegerPositiveLiteral
+: IntegerPositiveLiteral
+| IntegerNegativeLiteral
;
//
Modified: cassandra/trunk/test/unit/org/apache/cassandra/cli/CliTest.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/test/unit/org/apache/cassandra/cli/CliTest.java?rev=1087404&r1=1087403&r2=1087404&view=diff
==
--- cassandra/trunk/test/unit/org/apache/cassandra/cli/CliTest.java (original)
+++ cassandra/trunk/test/unit/org/apache/cassandra/cli/CliTest.java Thu Mar 31
18:39:37 2011
@@ -40,7 +40,11 @@ public class CliTest extends CleanupHelp
"create column family CF1 with comparator=UTF8Type and
column_metadata=[{ column_name:world, validation_class:IntegerType,
index_type:0, index_name:IdxName }, { column_name:world2,
validation_class:LongType, index_type:KEYS, index_name:LongIdxName}];",
"assume CF1 keys as utf8;",
"set CF1[hello][world] = 123848374878933948398384;",
+"set CF1[hello][-31337] = 'some string value';",
+"get CF1[hello][-31337];",
"get CF1[hello][world];",
+"set CF1[hello][-31337] = -23876;",
+"set CF1[hello][-31337] = long(-23876);",
"set CF1[hello][world2] = 15;",
"get CF1 where world2 = long(15);",
"get cF1 where world2 = long(15);",
@@ -79,6 +83,10 @@ public class CliTest extends CleanupHelp
"get SCF1['hello'][1][] as Long;",
"get SCF1['hello'][1][] as Bytes;",
"set SCF1['hello'][1][] = Long(1234);",
+"set SCF1['hello'][-1][-12] = Long(5678);",
+"get SCF1['hello'][-1][-12];",
+"set SCF1['hello'][-1][-12] = -340897;",
+"set SCF1['hello'][-1][-12] = integer(-340897);",
"get SCF1['hello'][1][];",
"get SCF1['hello'][1][] as Long;",
"del SCF1['hello'][1][];",
svn commit: r1087403 - in /cassandra/trunk: ./ contrib/ contrib/stress/src/org/apache/cassandra/contrib/stress/ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/db/
Author: jbellis
Date: Thu Mar 31 18:38:10 2011
New Revision: 1087403
URL: http://svn.apache.org/viewvc?rev=1087403&view=rev
Log:
merge from 0.7
Modified:
cassandra/trunk/ (props changed)
cassandra/trunk/CHANGES.txt
cassandra/trunk/contrib/ (props changed)
cassandra/trunk/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
(props changed)
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java
(props changed)
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java
(props changed)
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java
(props changed)
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java
(props changed)
cassandra/trunk/src/java/org/apache/cassandra/db/migration/DropColumnFamily.java
cassandra/trunk/src/java/org/apache/cassandra/db/migration/DropKeyspace.java
cassandra/trunk/test/unit/org/apache/cassandra/db/DefsTest.java
Propchange: cassandra/trunk/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Thu Mar 31 18:38:10 2011
@@ -1,5 +1,5 @@
/cassandra/branches/cassandra-0.6:922689-1052356,1052358-1053452,1053454,1053456-1081914,1083000
-/cassandra/branches/cassandra-0.7:1026516-1086415
+/cassandra/branches/cassandra-0.7:1026516-1087402
/cassandra/branches/cassandra-0.7.0:1053690-1055654
/cassandra/tags/cassandra-0.7.0-rc3:1051699-1053689
/incubator/cassandra/branches/cassandra-0.3:774578-796573
Modified: cassandra/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=1087403&r1=1087402&r2=1087403&view=diff
==
--- cassandra/trunk/CHANGES.txt (original)
+++ cassandra/trunk/CHANGES.txt Thu Mar 31 18:38:10 2011
@@ -79,8 +79,7 @@
* fixes for cache save/load (CASSANDRA-2172, -2174)
* Handle whole-row deletions in CFOutputFormat (CASSANDRA-2014)
* Make memtable_flush_writers flush in parallel (CASSANDRA-2178)
- * make key cache preheating default to false; enable with
- -Dcompaction_preheat_key_cache=true (CASSANDRA-2175)
+ * Add compaction_preheat_key_cache option (CASSANDRA-2175)
* refactor stress.py to have only one copy of the format string
used for creating row keys (CASSANDRA-2108)
* validate index names for \w+ (CASSANDRA-2196)
Propchange: cassandra/trunk/contrib/
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Thu Mar 31 18:38:10 2011
@@ -1,5 +1,5 @@
/cassandra/branches/cassandra-0.6/contrib:922689-1052356,1052358-1053452,1053454,1053456-1068009
-/cassandra/branches/cassandra-0.7/contrib:1026516-1086415
+/cassandra/branches/cassandra-0.7/contrib:1026516-1087402
/cassandra/branches/cassandra-0.7.0/contrib:1053690-1055654
/cassandra/tags/cassandra-0.7.0-rc3/contrib:1051699-1053689
/incubator/cassandra/branches/cassandra-0.3/contrib:774578-796573
Modified:
cassandra/trunk/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
URL:
http://svn.apache.org/viewvc/cassandra/trunk/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java?rev=1087403&r1=1087402&r2=1087403&view=diff
==
---
cassandra/trunk/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
(original)
+++
cassandra/trunk/contrib/stress/src/org/apache/cassandra/contrib/stress/Session.java
Thu Mar 31 18:38:10 2011
@@ -119,6 +119,12 @@ public class Session
{
CommandLine cmd = parser.parse(availableOptions, arguments);
+if (cmd.getArgs().length > 0)
+{
+System.err.println("Application does not allow arbitrary
arguments: " + StringUtils.join(cmd.getArgList(), ", "));
+System.exit(1);
+}
+
if (cmd.hasOption("h"))
throw new IllegalArgumentException("help");
Propchange:
cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java
--
--- svn:mergeinfo (original)
+++ svn:mergeinfo Thu Mar 31 18:38:10 2011
@@ -1,5 +1,5 @@
/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-1052356,1052358-1053452,1053454,1053456-1081914,1083000
-/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1086415
+/cassandra/branches/cassandra-0.7/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:1026516-1087402
/cassandra/branches/cassandra-0.7.0/interface/thrift/gen-java/org/apache/cassandra/thrif
svn commit: r1087402 - in /cassandra/branches/cassandra-0.7: src/java/org/apache/cassandra/db/migration/DropColumnFamily.java src/java/org/apache/cassandra/db/migration/DropKeyspace.java test/unit/org
Author: jbellis
Date: Thu Mar 31 18:34:56 2011
New Revision: 1087402
URL: http://svn.apache.org/viewvc?rev=1087402&view=rev
Log:
snapshot must be performed before flushlock must be acquired, or we deadlock.
See #2381
patch by jbellis
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropColumnFamily.java
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropKeyspace.java
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/db/DefsTest.java
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropColumnFamily.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropColumnFamily.java?rev=1087402&r1=1087401&r2=1087402&view=diff
==
---
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropColumnFamily.java
(original)
+++
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropColumnFamily.java
Thu Mar 31 18:34:56 2011
@@ -80,11 +80,12 @@ public class DropColumnFamily extends Mi
if (!clientMode)
{
+cfs.snapshot(Table.getTimestampedSnapshotName(null));
+
CompactionManager.instance.getCompactionLock().lock();
cfs.flushLock.lock();
try
{
-cfs.snapshot(Table.getTimestampedSnapshotName(null));
Table.open(ksm.name).dropCf(cfm.cfId);
}
finally
Modified:
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropKeyspace.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropKeyspace.java?rev=1087402&r1=1087401&r2=1087402&view=diff
==
---
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropKeyspace.java
(original)
+++
cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/migration/DropKeyspace.java
Thu Mar 31 18:34:56 2011
@@ -63,10 +63,10 @@ public class DropKeyspace extends Migrat
CFMetaData.purge(cfm);
if (!clientMode)
{
+cfs.snapshot(snapshotName);
cfs.flushLock.lock();
try
{
-cfs.snapshot(snapshotName);
Table.open(ksm.name).dropCf(cfm.cfId);
}
finally
Modified:
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/db/DefsTest.java
URL:
http://svn.apache.org/viewvc/cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/db/DefsTest.java?rev=1087402&r1=1087401&r2=1087402&view=diff
==
---
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/db/DefsTest.java
(original)
+++
cassandra/branches/cassandra-0.7/test/unit/org/apache/cassandra/db/DefsTest.java
Thu Mar 31 18:34:56 2011
@@ -411,7 +411,7 @@ public class DefsTest extends CleanupHel
assert ks != null;
final CFMetaData cfm = ks.cfMetaData().get("Standard2");
assert cfm != null;
-
+
// write some data, force a flush, then verify that files exist on
disk.
RowMutation rm = new RowMutation(ks.name, dk.key);
for (int i = 0; i < 100; i++)
@@ -451,7 +451,28 @@ public class DefsTest extends CleanupHel
assert th instanceof NullPointerException;
}
}
-
+
+@Test
+public void dropKSUnflushed() throws ConfigurationException, IOException,
ExecutionException, InterruptedException
+{
+DecoratedKey dk = Util.dk("dropKs");
+// sanity
+final KSMetaData ks =
DatabaseDescriptor.getTableDefinition("Keyspace3");
+assert ks != null;
+final CFMetaData cfm = ks.cfMetaData().get("Standard1");
+assert cfm != null;
+
+// write some data
+RowMutation rm = new RowMutation(ks.name, dk.key);
+for (int i = 0; i < 100; i++)
+rm.add(new QueryPath(cfm.cfName, null, ByteBufferUtil.bytes(("col"
+ i))), ByteBufferUtil.bytes("anyvalue"), 1L);
+rm.apply();
+
+new DropKeyspace(ks.name).apply();
+
+assert DatabaseDescriptor.getTableDefinition(ks.name) == null;
+}
+
@Test
public void renameKs() throws ConfigurationException, IOException,
ExecutionException, InterruptedException
{
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014128#comment-13014128 ] Pavel Yaskevich commented on CASSANDRA-1902: I have an idea of how to avoid mmap at all at SSTableIdentityIterator by using SSTableReader.dfile which is already mmaped in segments, it should lower i/o pressure during compaction with migration ON. > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 0.7.5, 0.8 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2409) Add INSERT support to CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reassigned CASSANDRA-2409: - Assignee: Pavel Yaskevich > Add INSERT support to CQL > - > > Key: CASSANDRA-2409 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2409 > Project: Cassandra > Issue Type: Bug > Components: API >Reporter: Jonathan Ellis >Assignee: Pavel Yaskevich > Fix For: 0.8 > > > There are two reasons to support INSERT: > - It helps new users feel comfortable (everyone's first statement will be to > try to INSERT something, we should make that a positive experience instead of > slapping them) > - Even though it is synonymous with update it is still useful in your code to > have both to help communicate intent, similar to choosing good variable names > The only downside is explaining how INSERT isn't a "true" insert because it > doesn't error out if the row already exists -- but we already have to explain > that same concept for UPDATE; the cognitive load is extremely minor. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-2409) Add INSERT support to CQL
Add INSERT support to CQL - Key: CASSANDRA-2409 URL: https://issues.apache.org/jira/browse/CASSANDRA-2409 Project: Cassandra Issue Type: Bug Components: API Reporter: Jonathan Ellis Fix For: 0.8 There are two reasons to support INSERT: - It helps new users feel comfortable (everyone's first statement will be to try to INSERT something, we should make that a positive experience instead of slapping them) - Even though it is synonymous with update it is still useful in your code to have both to help communicate intent, similar to choosing good variable names The only downside is explaining how INSERT isn't a "true" insert because it doesn't error out if the row already exists -- but we already have to explain that same concept for UPDATE; the cognitive load is extremely minor. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2265) Fix distributed tests after updating Guava to 08
[
https://issues.apache.org/jira/browse/CASSANDRA-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Yaskevich updated CASSANDRA-2265:
---
Attachment: CASSANDRA-2265.patch
Moved to Whirr 0.4.0, changed README.txt to reflect changes + added generation
of the checksum for distribution file (instance setup won't work without it).
> Fix distributed tests after updating Guava to 08
>
>
> Key: CASSANDRA-2265
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2265
> Project: Cassandra
> Issue Type: Sub-task
> Components: Core
>Affects Versions: 0.8
>Reporter: Pavel Yaskevich
>Assignee: Pavel Yaskevich
>Priority: Blocker
> Fix For: 0.8
>
> Attachments: 2265-partial.diff, CASSANDRA-2265.patch
>
>
> Distributed tests fail to start cluster:
> {code}
> [junit] java.lang.NoSuchMethodError:
> com.google.common.net.InternetDomainName.isValid(Ljava/lang/String;)Z
> [junit] at
> org.jclouds.rest.binders.BindAsHostPrefix.bindToRequest(BindAsHostPrefix.java:52)
> [junit] at
> org.jclouds.aws.s3.binders.BindAsHostPrefixIfConfigured.bindToRequest(BindAsHostPrefixIfConfigured.java:67)
> [junit] at
> org.jclouds.rest.internal.RestAnnotationProcessor.decorateRequest(RestAnnotationProcessor.java:860)
> [junit] at
> org.jclouds.rest.internal.RestAnnotationProcessor.createRequest(RestAnnotationProcessor.java:477)
> [junit] at
> org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFuture(AsyncRestClientProxy.java:117)
> [junit] at
> org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:97)
> [junit] at $Proxy58.objectExists(Unknown Source)
> [junit] at
> org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:134)
> [junit] at $Proxy59.objectExists(Unknown Source)
> [junit] at
> org.jclouds.aws.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:184)
> [junit] at
> org.jclouds.blobstore.internal.BaseBlobMap.containsKey(BaseBlobMap.java:201)
> [junit] at
> org.jclouds.blobstore.internal.InputStreamMapImpl.putInternal(InputStreamMapImpl.java:187)
> [junit] at
> org.jclouds.blobstore.internal.InputStreamMapImpl.putFile(InputStreamMapImpl.java:166)
> [junit] at
> org.apache.cassandra.utils.BlobUtils.storeBlob(BlobUtils.java:85)
> [junit] at
> org.apache.cassandra.CassandraServiceController.startup(CassandraServiceController.java:169)
> [junit] at
> org.apache.cassandra.CassandraServiceController.ensureClusterRunning(CassandraServiceController.java:236)
> [junit] at org.apache.cassandra.TestBase.setUp(TestBase.java:140)
> [junit] at
> org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:134)
> [junit] at $Proxy59.objectExists(Unknown Source)
> [junit] at
> org.jclouds.aws.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:184)
> [junit] at
> org.jclouds.blobstore.internal.BaseBlobMap.containsKey(BaseBlobMap.java:201)
> [junit] at
> org.jclouds.blobstore.internal.InputStreamMapImpl.putInternal(InputStreamMapImpl.java:187)
> [junit] at
> org.jclouds.blobstore.internal.InputStreamMapImpl.putFile(InputStreamMapImpl.java:166)
> [junit] at
> org.apache.cassandra.utils.BlobUtils.storeBlob(BlobUtils.java:85)
> [junit] at
> org.apache.cassandra.CassandraServiceController.startup(CassandraServiceController.java:169)
> [junit] at
> org.apache.cassandra.CassandraServiceController.ensureClusterRunning(CassandraServiceController.java:236)
> [junit] at org.apache.cassandra.TestBase.setUp(TestBase.java:140)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2231) Add CompositeType comparer to the comparers provided in org.apache.cassandra.db.marshal
[ https://issues.apache.org/jira/browse/CASSANDRA-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014101#comment-13014101 ] Sylvain Lebresne commented on CASSANDRA-2231: - It's a good idea. I'll update the patch. > Add CompositeType comparer to the comparers provided in > org.apache.cassandra.db.marshal > --- > > Key: CASSANDRA-2231 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2231 > Project: Cassandra > Issue Type: Improvement > Components: Contrib >Affects Versions: 0.7.3 >Reporter: Ed Anuff >Assignee: Sylvain Lebresne >Priority: Minor > Fix For: 0.7.5 > > Attachments: CompositeType-and-DynamicCompositeType.patch, > edanuff-CassandraCompositeType-1e253c4.zip > > > CompositeType is a custom comparer that makes it possible to create > comparable composite values out of the basic types that Cassandra currently > supports, such as Long, UUID, etc. This is very useful in both the creation > of custom inverted indexes using columns in a skinny row, where each column > name is a composite value, and also when using Cassandra's built-in secondary > index support, where it can be used to encode the values in the columns that > Cassandra indexes. One scenario for the usage of these is documented here: > http://www.anuff.com/2010/07/secondary-indexes-in-cassandra.html. Source for > contribution is attached and has been previously maintained on github here: > https://github.com/edanuff/CassandraCompositeType -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014001#comment-13014001 ] Pavel Yaskevich edited comment on CASSANDRA-1902 at 3/31/11 3:13 PM: - What is the difference with the current code? was (Author: xedin): What is the different with the current code? > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 0.7.5, 0.8 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014001#comment-13014001 ] Pavel Yaskevich commented on CASSANDRA-1902: What is the different with the current code? > Migrate cached pages during compaction > --- > > Key: CASSANDRA-1902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1902 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.1 >Reporter: T Jake Luciani >Assignee: Pavel Yaskevich > Fix For: 0.7.5, 0.8 > > Attachments: > 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt, > 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt, > 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt, > CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch, > CASSANDRA-1902-v6.patch > > Original Estimate: 32h > Time Spent: 56h > Remaining Estimate: 0h > > Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a > pre-compacted CF during the compaction process. This is now important since > CASSANDRA-1470 caches effectively nothing. > For example an active CF being compacted hurts reads since nothing is cached > in the new SSTable. > The purpose of this ticket then is to make sure SOME data is cached from > active CFs. This can be done my monitoring which Old SSTables are in the page > cache and caching active rows in the New SStable. > A simpler yet similar approach is described here: > http://insights.oetiker.ch/linux/fadvise/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1902) Migrate cached pages during compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013996#comment-13013996
]
T Jake Luciani commented on CASSANDRA-1902:
---
Here's my test results:
1. Loaded in just enough to fit in page cache + flushed
2. Primed reads
3. Overwrite with same data + flushed
4. Ran major compaction while reading as much as possible the entire time.
Read results for 100k during major compaction (migration ON):
{noformat}
19,13,14,14,82,16,15,67,19,14,47,53,74,15,14,46,32,15,13,65,15,14,67,16,13,59,12,11,11
^^^ ^ ^^ ^^^
{noformat}
vs with migration OFF
~14-16 X times, then 260, 68, 16
> Migrate cached pages during compaction
> ---
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.7.1
>Reporter: T Jake Luciani
>Assignee: Pavel Yaskevich
> Fix For: 0.7.5, 0.8
>
> Attachments:
> 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt,
> 1902-BufferedSegmentedFile-logandsleep.txt, 1902-formatted.txt,
> 1902-per-column-migration-rebase2.txt, 1902-per-column-migration.txt,
> CASSANDRA-1902-v3.patch, CASSANDRA-1902-v4.patch, CASSANDRA-1902-v5.patch,
> CASSANDRA-1902-v6.patch
>
> Original Estimate: 32h
> Time Spent: 56h
> Remaining Estimate: 0h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a
> pre-compacted CF during the compaction process. This is now important since
> CASSANDRA-1470 caches effectively nothing.
> For example an active CF being compacted hurts reads since nothing is cached
> in the new SSTable.
> The purpose of this ticket then is to make sure SOME data is cached from
> active CFs. This can be done my monitoring which Old SSTables are in the page
> cache and caching active rows in the New SStable.
> A simpler yet similar approach is described here:
> http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2404) if out of disk space reclaim compacted SSTables during memtable flush
[ https://issues.apache.org/jira/browse/CASSANDRA-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013908#comment-13013908 ] Stu Hood commented on CASSANDRA-2404: - Actually, I would be all for _moving_ the logic from getDataFileLocation to createFlushWriter... not having room to flush is much more urgent than not having room to compact, and as a node gets closer to full, large bucket compactions will begin to spam GC for compactions that aren't going to be possible anyway. > if out of disk space reclaim compacted SSTables during memtable flush > - > > Key: CASSANDRA-2404 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2404 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.7.4 >Reporter: Aaron Morton >Assignee: Aaron Morton >Priority: Minor > Fix For: 0.7.5 > > > During compaction if there is not enough disk space we invoke GC to reclaim > unused space. > During memtable and binary memtable flush we just error out if there is not > enough disk space to flush the table. > Can we make cfs.createFlushWriter() use the same logic as > Table.getDataFileLocation() to reclaim space if needed? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2156) Compaction Throttling
[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013867#comment-13013867
]
Stu Hood edited comment on CASSANDRA-2156 at 3/31/11 10:07 AM:
---
1. Fixed
2. Added and used {{FileUtils.close(Collection)}}
3. targetBytesPerMS only changes when the number of active threads changes: it
leads to nice (imo) periodic feedback of running compactions in the log when
compactions start or finish
4. Assuming compaction multithreading makes it in, throttling should never be
disabled... for someone who really wants to disable it, setting it to a high
enough value that it never kicks in should be sufficient?
5. Maybe... but dynamically adjusting the frequency at which we throttle and
update {{bytesRead}} would probably be better to do in another thread?
Regarding the approach to setting compaction_throughput_mb_per_sec: each bucket
probably contains {{MIN_THRESHOLD}} times more data than the previous bucket,
and needs to be compacted {{1 / MIN_THRESHOLD}} times as often (see the math in
the description). This means that the number of buckets influences how fast you
need to compact, and that each additional bucket adds a linear amount of
necessary throughput (+ 1x your flush rate). Therefore, if you have 15 bucket
levels, and you are flushing {{1 MB/s}}, you need to compact at {{1 MB/s * 15}}.
As an example: with {{MIN_THRESHOLD=2}}, each bucket is twice is large as the
previous. Say that we have 4 levels (buckets of sizes 1, 2, 4, 8) and that we
need a compaction in the largest bucket. The amount of data that needs to be
compacted in that bucket will be equal to 1 more than the sum of the sizes of
all the other buckets (1 + 2 + 4 == 8 - 1). So, ideally we would be able to
compact those 8 units in _exactly_ the time it takes for 1 more unit to be
flushed, and for the compactions of the other buckets to trickle up and refill
the largest bucket. Pheew?
CASSANDRA-2171 will allow us to calculate the flush rate, which we can then
multiply by the count of buckets (note... one tiny missing piece is determining
how many buckets are "empty": an empty bucket is not created in the current
approach).
> Final question. Would it be better to have fewer parallel compactions
As a base case, with no parallelism at all, you _will_ fall behind on
compaction, because every new bucket is a chance to compact. It's a fundamental
question, but I haven't thought about it... sorry.
was (Author: stuhood):
1. Fixed
2. Added and used {{FileUtils.close(Collection)}}
3. targetBytesPerMS only changes when the number of active threads changes: it
leads to nice (imo) periodic feedback of running compactions in the log when
compactions start or finish
4. Assuming compaction multithreading makes it in, throttling should never be
disabled... for someone who really wants to disable it, setting it to a high
enough value that it never kicks in should be sufficient?
5. Maybe... but dynamically adjusting the frequency at which we throttle and
update {{bytesRead}} would probably be better to do in another thread?
Regarding the approach to setting compaction_throughput_mb_per_sec: each bucket
probably contains {{MIN_THRESHOLD}} times more data than the previous bucket,
and needs to be compacted {{1 / MIN_THRESHOLD}} times as often (see the math in
the description). This means that the number of buckets influences how fast you
need to compact, and that each additional bucket adds a linear amount of
necessary throughput (+ 1x your flush rate). Therefore, if you have 15 bucket
levels, and you are flushing {{1 MB/s}}, you need to compact at {{1 MB/s * 15}}.
As an example: with {{MIN_THRESHOLD=2}}, each bucket is twice is large as the
previous. Say that we have 4 levels (buckets of sizes 1, 2, 4, 8) and that we
need a compaction in the largest bucket. The amount of data that needs to be
compacted in that bucket will be equal to 1 more than the sum of the sizes of
all the other buckets (1 + 2 + 4 == 8 - 1). So, ideally we would be able to
compact those 8 units in the time it _exactly_ the time it takes for 1 more
unit to be flushed, and for the compactions of the other buckets to trickle up
and refill the largest bucket. Pheew?
CASSANDRA-2171 will allow us to calculate the flush rate, which we can then
multiply by the count of buckets (note... one tiny missing piece is determining
how many buckets are "empty": an empty bucket is not created in the current
approach).
> Final question. Would it be better to have fewer parallel compactions
As a base case, with no parallelism at all, you _will_ fall behind on
compaction, because every new bucket is a chance to compact. It's a fundamental
question, but I haven't thought about it... sorry.
> Compaction Throttling
> -
>
> Key: CASSANDRA-2156
>
[jira] [Issue Comment Edited] (CASSANDRA-2156) Compaction Throttling
[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013867#comment-13013867
]
Stu Hood edited comment on CASSANDRA-2156 at 3/31/11 10:08 AM:
---
1. Fixed
2. Added and used {{FileUtils.close(Collection)}}
3. targetBytesPerMS only changes when the number of active threads changes: it
leads to nice (imo) periodic feedback of running compactions in the log when
compactions start or finish
4. Assuming compaction multithreading makes it in, throttling should never be
disabled... for someone who really wants to disable it, setting it to a high
enough value that it never kicks in should be sufficient?
5. Maybe... but dynamically adjusting the frequency at which we throttle and
update {{bytesRead}} would probably be better to do in another ticket?
Regarding the approach to setting compaction_throughput_mb_per_sec: each bucket
probably contains {{MIN_THRESHOLD}} times more data than the previous bucket,
and needs to be compacted {{1 / MIN_THRESHOLD}} times as often (see the math in
the description). This means that the number of buckets influences how fast you
need to compact, and that each additional bucket adds a linear amount of
necessary throughput (+ 1x your flush rate). Therefore, if you have 15 bucket
levels, and you are flushing {{1 MB/s}}, you need to compact at {{1 MB/s * 15}}.
As an example: with {{MIN_THRESHOLD=2}}, each bucket is twice is large as the
previous. Say that we have 4 levels (buckets of sizes 1, 2, 4, 8) and that we
need a compaction in the largest bucket. The amount of data that needs to be
compacted in that bucket will be equal to 1 more than the sum of the sizes of
all the other buckets (1 + 2 + 4 == 8 - 1). So, ideally we would be able to
compact those 8 units in _exactly_ the time it takes for 1 more unit to be
flushed, and for the compactions of the other buckets to trickle up and refill
the largest bucket. Pheew?
CASSANDRA-2171 will allow us to calculate the flush rate, which we can then
multiply by the count of buckets (note... one tiny missing piece is determining
how many buckets are "empty": an empty bucket is not created in the current
approach).
> Final question. Would it be better to have fewer parallel compactions
As a base case, with no parallelism at all, you _will_ fall behind on
compaction, because every new bucket is a chance to compact. It's a fundamental
question, but I haven't thought about it... sorry.
was (Author: stuhood):
1. Fixed
2. Added and used {{FileUtils.close(Collection)}}
3. targetBytesPerMS only changes when the number of active threads changes: it
leads to nice (imo) periodic feedback of running compactions in the log when
compactions start or finish
4. Assuming compaction multithreading makes it in, throttling should never be
disabled... for someone who really wants to disable it, setting it to a high
enough value that it never kicks in should be sufficient?
5. Maybe... but dynamically adjusting the frequency at which we throttle and
update {{bytesRead}} would probably be better to do in another thread?
Regarding the approach to setting compaction_throughput_mb_per_sec: each bucket
probably contains {{MIN_THRESHOLD}} times more data than the previous bucket,
and needs to be compacted {{1 / MIN_THRESHOLD}} times as often (see the math in
the description). This means that the number of buckets influences how fast you
need to compact, and that each additional bucket adds a linear amount of
necessary throughput (+ 1x your flush rate). Therefore, if you have 15 bucket
levels, and you are flushing {{1 MB/s}}, you need to compact at {{1 MB/s * 15}}.
As an example: with {{MIN_THRESHOLD=2}}, each bucket is twice is large as the
previous. Say that we have 4 levels (buckets of sizes 1, 2, 4, 8) and that we
need a compaction in the largest bucket. The amount of data that needs to be
compacted in that bucket will be equal to 1 more than the sum of the sizes of
all the other buckets (1 + 2 + 4 == 8 - 1). So, ideally we would be able to
compact those 8 units in _exactly_ the time it takes for 1 more unit to be
flushed, and for the compactions of the other buckets to trickle up and refill
the largest bucket. Pheew?
CASSANDRA-2171 will allow us to calculate the flush rate, which we can then
multiply by the count of buckets (note... one tiny missing piece is determining
how many buckets are "empty": an empty bucket is not created in the current
approach).
> Final question. Would it be better to have fewer parallel compactions
As a base case, with no parallelism at all, you _will_ fall behind on
compaction, because every new bucket is a chance to compact. It's a fundamental
question, but I haven't thought about it... sorry.
> Compaction Throttling
> -
>
> Key: CASSANDRA-2156
>
[jira] [Issue Comment Edited] (CASSANDRA-2156) Compaction Throttling
[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013867#comment-13013867
]
Stu Hood edited comment on CASSANDRA-2156 at 3/31/11 10:06 AM:
---
1. Fixed
2. Added and used {{FileUtils.close(Collection)}}
3. targetBytesPerMS only changes when the number of active threads changes: it
leads to nice (imo) periodic feedback of running compactions in the log when
compactions start or finish
4. Assuming compaction multithreading makes it in, throttling should never be
disabled... for someone who really wants to disable it, setting it to a high
enough value that it never kicks in should be sufficient?
5. Maybe... but dynamically adjusting the frequency at which we throttle and
update {{bytesRead}} would probably be better to do in another thread?
Regarding the approach to setting compaction_throughput_mb_per_sec: each bucket
probably contains {{MIN_THRESHOLD}} times more data than the previous bucket,
and needs to be compacted {{1 / MIN_THRESHOLD}} times as often (see the math in
the description). This means that the number of buckets influences how fast you
need to compact, and that each additional bucket adds a linear amount of
necessary throughput (+ 1x your flush rate). Therefore, if you have 15 bucket
levels, and you are flushing {{1 MB/s}}, you need to compact at {{1 MB/s * 15}}.
As an example: with {{MIN_THRESHOLD=2}}, each bucket is twice is large as the
previous. Say that we have 4 levels (buckets of sizes 1, 2, 4, 8) and that we
need a compaction in the largest bucket. The amount of data that needs to be
compacted in that bucket will be equal to 1 more than the sum of the sizes of
all the other buckets (1 + 2 + 4 == 8 - 1). So, ideally we would be able to
compact those 8 units in the time it _exactly_ the time it takes for 1 more
unit to be flushed, and for the compactions of the other buckets to trickle up
and refill the largest bucket. Pheew?
CASSANDRA-2171 will allow us to calculate the flush rate, which we can then
multiply by the count of buckets (note... one tiny missing piece is determining
how many buckets are "empty": an empty bucket is not created in the current
approach).
> Final question. Would it be better to have fewer parallel compactions
As a base case, with no parallelism at all, you _will_ fall behind on
compaction, because every new bucket is a chance to compact. It's a fundamental
question, but I haven't thought about it... sorry.
was (Author: stuhood):
1. Fixed
2. Added and used {{FileUtils.close(Collection)}}
3. targetBytesPerMS only changes when the number of active threads changes: it
leads to nice (imo) periodic feedback of running compactions in the log when
compactions start or finish
4. Assuming compaction multithreading makes it in, throttling should never be
disabled... for someone who really wants to disable it, setting it to a high
enough value that it never kicks in should be sufficient?
5. Maybe... but dynamically adjusting the frequency at which we throttle and
update {{bytesRead}} would probably be better to do in another thread?
Regarding the approach to setting compaction_throughput_mb_per_sec: each bucket
probably contains {{MIN_THRESHOLD}} times more data than the previous bucket,
and needs to be compacted {{1 / MIN_THRESHOLD}} times as often (see the math in
the description). This means that the number of buckets influences how fast you
need to compact, and that each additional bucket adds a linear amount of
necessary throughput (+ 1x your flush rate). Therefore, if you have 15 bucket
levels, and you are flushing {{1 MB/s}}, you need to compact at {{1 MB/s * 15}}.
As an example: with {{MIN_THRESHOLD=2}}, each bucket is twice is large as the
previous. Say that we have 4 levels (buckets of sizes 1, 2, 4, 8) and that we
need a compaction in the largest bucket. The amount of data that needs to be
compacted in that bucket will be equal to 1 less than the sum of the sizes of
all the other buckets (1 + 2 + 4 == 8 - 1). So, ideally we would be able to
compact those 8 units in the time it _exactly_ the time it takes for 1 more
unit to be flushed, and for the compactions of the other buckets to trickle up
and refill the largest bucket. Pheew?
CASSANDRA-2171 will allow us to calculate the flush rate, which we can then
multiply by the count of buckets (note... one tiny missing piece is determining
how many buckets are "empty": an empty bucket is not created in the current
approach).
> Final question. Would it be better to have fewer parallel compactions
As a base case, with no parallelism at all, you _will_ fall behind on
compaction, because every new bucket is a chance to compact. It's a fundamental
question, but I haven't thought about it... sorry.
> Compaction Throttling
> -
>
> Key: CASSANDR
[jira] [Created] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.) -- Key: CASSANDRA-2408 URL: https://issues.apache.org/jira/browse/CASSANDRA-2408 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 0.7.4 Reporter: Thibaut Fix For: 0.7.5 Hi, We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing the compactation to fail. We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours. A faulty node shouldn't affect the key ordering of adjacent nodes? INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')] INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f39323237303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838) INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f39323237303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534) INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.io.IOException: Keys must be written in ascending order. at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2408) Faulty memory causes adjacent nodes to have invalid data and fail compactation (java.io.IOException: Keys must be written in ascending order.)
[ https://issues.apache.org/jira/browse/CASSANDRA-2408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thibaut updated CASSANDRA-2408: --- Description: Hi, We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing their compactation to fail. We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours. A faulty node shouldn't affect the key ordering of adjacent nodes? INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')] INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f39323237303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838) INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 109) Current key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988136_17070303-ab24-4207-83bd-1604e2c159e4, 62613537386265643565373561303664313135366461626363633638616264355f772e6b677475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f39323237303733353431333938383133365f31373037303330332d616232342d343230372d383362642d313630346532633135396534) INFO [CompactionExecutor:1] 2011-03-31 09:58:26,515 SSTableWriter.java (line 110) Writing into file /cassandra/data/table_lists/table_lists-tmp-f-1861-Data.db ERROR [CompactionExecutor:1] 2011-03-31 09:58:26,516 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.io.IOException: Keys must be written in ascending order. at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:111) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:128) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:452) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:124) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:94) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) was: Hi, We had to replace a node with faulty ram. Besides the cassandra cluster getting unresponsive, we also observerd a "keys must be written in ascending order" exception on all adjacent quorum nodes, causing the compactation to fail. We had another node with faulty ram a week earlier, and it was causing the same errors on it's neighbours. A faulty node shouldn't affect the key ordering of adjacent nodes? INFO [CompactionExecutor:1] 2011-03-31 09:57:29,529 CompactionManager.java (line 396) Compacting [SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1793-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1798-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1803-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1808-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1832-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1841-Data.db'),SSTableReader(path='/cassandra/data/table_lists/table_lists-f-1852-Data.db')] INFO [CompactionExecutor:1] 2011-03-31 09:58:26,437 SSTableWriter.java (line 108) Last written key : DecoratedKey(ba578bed5e75a06d1156dabccc68abd5_www.abc.com_hostinlinks_9223370735413988125_870cc25d-7e4f-437b-aa83-a311dfbf7f88, 62613537386265643565373561303664313135366461626363633638616264355f772e6b6f7475736f7a6c756b2e636f6d5f686f7374696e6c696e6b735f39323237303733353431333938383132355f38373063633235642d376534662d343337622d616138332d613331316466626637663838) INFO [CompactionExecutor:1] 2011-03-31
[jira] [Updated] (CASSANDRA-2191) Multithread across compaction buckets
[ https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stu Hood updated CASSANDRA-2191: Attachment: 0004-Try-harder-to-close-scanners-in-compaction-close.txt 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt 0001-Add-a-compacting-set-to-DataTracker.txt > Multithread across compaction buckets > - > > Key: CASSANDRA-2191 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2191 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Stu Hood >Priority: Critical > Labels: compaction > Fix For: 0.8 > > Attachments: 0001-Add-a-compacting-set-to-DataTracker.txt, > 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt, > 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt, > 0004-Try-harder-to-close-scanners-in-compaction-close.txt > > > This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and > reasoning are different enough to open a separate issue. > The problem with compactions currently is that they compact the set of > sstables that existed the moment the compaction started. This means that for > longer running compactions (even when running as fast as possible on the > hardware), a very large number of new sstables might be created in the > meantime. We have observed this proliferation of sstables killing performance > during major/high-bucketed compactions. > One approach would be to pause compactions in upper buckets (containing > larger files) when compactions in lower buckets become possible. While this > would likely solve the problem with read performance, it does not actually > help us perform compaction any faster, which is a reasonable requirement for > other situations. > Instead, we need to be able to perform any compactions that are currently > required in parallel, independent of what bucket they might be in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2156) Compaction Throttling
[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stu Hood updated CASSANDRA-2156:
Attachment: 0005-Throttle-total-compaction-to-a-configurable-throughput.txt
> Compaction Throttling
> -
>
> Key: CASSANDRA-2156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2156
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Stu Hood
> Fix For: 0.8
>
> Attachments:
> 0005-Throttle-total-compaction-to-a-configurable-throughput.txt,
> for-0.6-0001-Throttle-compaction-to-a-fixed-throughput.txt,
> for-0.6-0002-Make-compaction-throttling-configurable.txt
>
>
> Compaction is currently relatively bursty: we compact as fast as we can, and
> then we wait for the next compaction to be possible ("hurry up and wait").
> Instead, to properly amortize compaction, you'd like to compact exactly as
> fast as you need to to keep the sstable count under control.
> For every new level of compaction, you need to increase the rate that you
> compact at: a rule of thumb that we're testing on our clusters is to
> determine the maximum number of buckets a node can support (aka, if the 15th
> bucket holds 750 GB, we're not going to have more than 15 buckets), and then
> multiply the flush throughput by the number of buckets to get a minimum
> compaction throughput to maintain your sstable count.
> Full explanation: for a min compaction threshold of {{T}}, the bucket at
> level {{N}} can contain {{SsubN = T^N}} 'units' (unit == memtable's worth of
> data on disk). Every time a new unit is added, it has a {{1/SsubN}} chance of
> causing the bucket at level N to fill. If the bucket at level N fills, it
> causes {{SsubN}} units to be compacted. So, for each active level in your
> system you have {{SubN * 1 / SsubN}}, or {{1}} amortized unit to compact any
> time a new unit is added.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2191) Multithread across compaction buckets
[ https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stu Hood updated CASSANDRA-2191: Attachment: (was: 0001-Add-a-compacting-set-to-sstabletracker.txt) > Multithread across compaction buckets > - > > Key: CASSANDRA-2191 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2191 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Stu Hood >Priority: Critical > Labels: compaction > Fix For: 0.8 > > > This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and > reasoning are different enough to open a separate issue. > The problem with compactions currently is that they compact the set of > sstables that existed the moment the compaction started. This means that for > longer running compactions (even when running as fast as possible on the > hardware), a very large number of new sstables might be created in the > meantime. We have observed this proliferation of sstables killing performance > during major/high-bucketed compactions. > One approach would be to pause compactions in upper buckets (containing > larger files) when compactions in lower buckets become possible. While this > would likely solve the problem with read performance, it does not actually > help us perform compaction any faster, which is a reasonable requirement for > other situations. > Instead, we need to be able to perform any compactions that are currently > required in parallel, independent of what bucket they might be in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2191) Multithread across compaction buckets
[ https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stu Hood updated CASSANDRA-2191: Attachment: (was: 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt) > Multithread across compaction buckets > - > > Key: CASSANDRA-2191 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2191 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Stu Hood >Priority: Critical > Labels: compaction > Fix For: 0.8 > > > This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and > reasoning are different enough to open a separate issue. > The problem with compactions currently is that they compact the set of > sstables that existed the moment the compaction started. This means that for > longer running compactions (even when running as fast as possible on the > hardware), a very large number of new sstables might be created in the > meantime. We have observed this proliferation of sstables killing performance > during major/high-bucketed compactions. > One approach would be to pause compactions in upper buckets (containing > larger files) when compactions in lower buckets become possible. While this > would likely solve the problem with read performance, it does not actually > help us perform compaction any faster, which is a reasonable requirement for > other situations. > Instead, we need to be able to perform any compactions that are currently > required in parallel, independent of what bucket they might be in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2156) Compaction Throttling
[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stu Hood updated CASSANDRA-2156:
Attachment: (was:
0001-Throttle-total-compaction-to-a-configurable-throughput.txt)
> Compaction Throttling
> -
>
> Key: CASSANDRA-2156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2156
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Stu Hood
> Fix For: 0.8
>
> Attachments:
> for-0.6-0001-Throttle-compaction-to-a-fixed-throughput.txt,
> for-0.6-0002-Make-compaction-throttling-configurable.txt
>
>
> Compaction is currently relatively bursty: we compact as fast as we can, and
> then we wait for the next compaction to be possible ("hurry up and wait").
> Instead, to properly amortize compaction, you'd like to compact exactly as
> fast as you need to to keep the sstable count under control.
> For every new level of compaction, you need to increase the rate that you
> compact at: a rule of thumb that we're testing on our clusters is to
> determine the maximum number of buckets a node can support (aka, if the 15th
> bucket holds 750 GB, we're not going to have more than 15 buckets), and then
> multiply the flush throughput by the number of buckets to get a minimum
> compaction throughput to maintain your sstable count.
> Full explanation: for a min compaction threshold of {{T}}, the bucket at
> level {{N}} can contain {{SsubN = T^N}} 'units' (unit == memtable's worth of
> data on disk). Every time a new unit is added, it has a {{1/SsubN}} chance of
> causing the bucket at level N to fill. If the bucket at level N fills, it
> causes {{SsubN}} units to be compacted. So, for each active level in your
> system you have {{SubN * 1 / SsubN}}, or {{1}} amortized unit to compact any
> time a new unit is added.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2191) Multithread across compaction buckets
[ https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stu Hood updated CASSANDRA-2191: Attachment: (was: 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt) > Multithread across compaction buckets > - > > Key: CASSANDRA-2191 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2191 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Stu Hood >Priority: Critical > Labels: compaction > Fix For: 0.8 > > > This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and > reasoning are different enough to open a separate issue. > The problem with compactions currently is that they compact the set of > sstables that existed the moment the compaction started. This means that for > longer running compactions (even when running as fast as possible on the > hardware), a very large number of new sstables might be created in the > meantime. We have observed this proliferation of sstables killing performance > during major/high-bucketed compactions. > One approach would be to pause compactions in upper buckets (containing > larger files) when compactions in lower buckets become possible. While this > would likely solve the problem with read performance, it does not actually > help us perform compaction any faster, which is a reasonable requirement for > other situations. > Instead, we need to be able to perform any compactions that are currently > required in parallel, independent of what bucket they might be in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2191) Multithread across compaction buckets
[
https://issues.apache.org/jira/browse/CASSANDRA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013837#comment-13013837
]
Stu Hood edited comment on CASSANDRA-2191 at 3/31/11 7:17 AM:
--
1. Added {{if (max < min || max < 1) return null;}} check
2. Switched to LinkedHashSet to preserve order
3. Fixed, thanks
4. The idea is that the throttling from CASSANDRA-2156 is a sufficient
preventative measure for this, but it will be important to enable it by
default, hopefully using the metrics from CASSANDRA-2171. I'd prefer not to
have two tunables, but it's worth discussing.
5. CompactionManagerMBean exposes getPendingTasks and getCompletedTasks with
entirely different meanings, so I'm not sure we'd gain anything by this
6. (see next comment)
7. Yes, it probably would, but I think that is an issue for a separate ticket.
Usually compacting the smallest bucket first (since the files are likely to be
hot in cache) is the biggest win (which we do): it will be very rare for higher
buckets to be more important
8. This would probably be a good idea, for example, if you have more than
{{max}} sstables in the minimum bucket and not enough active threads to
parallelize the bucket. Opened CASSANDRA-2407
9. I can't think of an easy way to do this, but if you can, I'm willing
was (Author: stuhood):
1. Added {{if (max < min || max < 0) return null;}} check
2. Switched to LinkedHashSet to preserve order
3. Fixed, thanks
4. The idea is that the throttling from CASSANDRA-2156 is a sufficient
preventative measure for this, but it will be important to enable it by
default, hopefully using the metrics from CASSANDRA-2171. I'd prefer not to
have two tunables, but it's worth discussing.
5. CompactionManagerMBean exposes getPendingTasks and getCompletedTasks with
entirely different meanings, so I'm not sure we'd gain anything by this
6. (see next comment)
7. Yes, it probably would, but I think that is an issue for a separate ticket.
Usually compacting the smallest bucket first (since the files are likely to be
hot in cache) is the biggest win (which we do): it will be very rare for higher
buckets to be more important
8. This would probably be a good idea, for example, if you have more than
{{max}} sstables in the minimum bucket and not enough active threads to
parallelize the bucket. Opened CASSANDRA-2407
9. I can't think of an easy way to do this, but if you can, I'm willing
> Multithread across compaction buckets
> -
>
> Key: CASSANDRA-2191
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2191
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Stu Hood
>Priority: Critical
> Labels: compaction
> Fix For: 0.8
>
> Attachments: 0001-Add-a-compacting-set-to-sstabletracker.txt,
> 0002-Use-the-compacting-set-of-sstables-to-schedule-multith.txt,
> 0003-Expose-multiple-compactions-via-JMX-and-deprecate-sing.txt
>
>
> This ticket overlaps with CASSANDRA-1876 to a degree, but the approaches and
> reasoning are different enough to open a separate issue.
> The problem with compactions currently is that they compact the set of
> sstables that existed the moment the compaction started. This means that for
> longer running compactions (even when running as fast as possible on the
> hardware), a very large number of new sstables might be created in the
> meantime. We have observed this proliferation of sstables killing performance
> during major/high-bucketed compactions.
> One approach would be to pause compactions in upper buckets (containing
> larger files) when compactions in lower buckets become possible. While this
> would likely solve the problem with read performance, it does not actually
> help us perform compaction any faster, which is a reasonable requirement for
> other situations.
> Instead, we need to be able to perform any compactions that are currently
> required in parallel, independent of what bucket they might be in.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2156) Compaction Throttling
[
https://issues.apache.org/jira/browse/CASSANDRA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013867#comment-13013867
]
Stu Hood commented on CASSANDRA-2156:
-
1. Fixed
2. Added and used {{FileUtils.close(Collection)}}
3. targetBytesPerMS only changes when the number of active threads changes: it
leads to nice (imo) periodic feedback of running compactions in the log when
compactions start or finish
4. Assuming compaction multithreading makes it in, throttling should never be
disabled... for someone who really wants to disable it, setting it to a high
enough value that it never kicks in should be sufficient?
5. Maybe... but dynamically adjusting the frequency at which we throttle and
update {{bytesRead}} would probably be better to do in another thread?
Regarding the approach to setting compaction_throughput_mb_per_sec: each bucket
probably contains {{MIN_THRESHOLD}} times more data than the previous bucket,
and needs to be compacted {{1 / MIN_THRESHOLD}} times as often (see the math in
the description). This means that the number of buckets influences how fast you
need to compact, and that each additional bucket adds a linear amount of
necessary throughput (+ 1x your flush rate). Therefore, if you have 15 bucket
levels, and you are flushing {{1 MB/s}}, you need to compact at {{1 MB/s * 15}}.
As an example: with {{MIN_THRESHOLD=2}}, each bucket is twice is large as the
previous. Say that we have 4 levels (buckets of sizes 1, 2, 4, 8) and that we
need a compaction in the largest bucket. The amount of data that needs to be
compacted in that bucket will be equal to 1 less than the sum of the sizes of
all the other buckets (1 + 2 + 4 == 8 - 1). So, ideally we would be able to
compact those 8 units in the time it _exactly_ the time it takes for 1 more
unit to be flushed, and for the compactions of the other buckets to trickle up
and refill the largest bucket. Pheew?
CASSANDRA-2171 will allow us to calculate the flush rate, which we can then
multiply by the count of buckets (note... one tiny missing piece is determining
how many buckets are "empty": an empty bucket is not created in the current
approach).
> Final question. Would it be better to have fewer parallel compactions
As a base case, with no parallelism at all, you _will_ fall behind on
compaction, because every new bucket is a chance to compact. It's a fundamental
question, but I haven't thought about it... sorry.
> Compaction Throttling
> -
>
> Key: CASSANDRA-2156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2156
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Stu Hood
> Fix For: 0.8
>
> Attachments:
> 0001-Throttle-total-compaction-to-a-configurable-throughput.txt,
> for-0.6-0001-Throttle-compaction-to-a-fixed-throughput.txt,
> for-0.6-0002-Make-compaction-throttling-configurable.txt
>
>
> Compaction is currently relatively bursty: we compact as fast as we can, and
> then we wait for the next compaction to be possible ("hurry up and wait").
> Instead, to properly amortize compaction, you'd like to compact exactly as
> fast as you need to to keep the sstable count under control.
> For every new level of compaction, you need to increase the rate that you
> compact at: a rule of thumb that we're testing on our clusters is to
> determine the maximum number of buckets a node can support (aka, if the 15th
> bucket holds 750 GB, we're not going to have more than 15 buckets), and then
> multiply the flush throughput by the number of buckets to get a minimum
> compaction throughput to maintain your sstable count.
> Full explanation: for a min compaction threshold of {{T}}, the bucket at
> level {{N}} can contain {{SsubN = T^N}} 'units' (unit == memtable's worth of
> data on disk). Every time a new unit is added, it has a {{1/SsubN}} chance of
> causing the bucket at level N to fill. If the bucket at level N fills, it
> causes {{SsubN}} units to be compacted. So, for each active level in your
> system you have {{SubN * 1 / SsubN}}, or {{1}} amortized unit to compact any
> time a new unit is added.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
