[cassandra-jdbc] push by wfs...@gmail.com - Fix for fix of JdbcRegressionTest for Issue 71 on 2013-04-20 15:22 GMT
Revision: efab20dcfee2
Author: Rick Shaw wfs...@gmail.com
Date: Sat Apr 20 08:22:54 2013
Log: Fix for fix of JdbcRegressionTest for Issue 71
http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/source/detail?r=efab20dcfee2
Modified:
/src/test/java/org/apache/cassandra/cql/jdbc/JdbcRegressionTest.java
===
--- /src/test/java/org/apache/cassandra/cql/jdbc/JdbcRegressionTest.java
Thu Apr 18 12:36:05 2013
+++ /src/test/java/org/apache/cassandra/cql/jdbc/JdbcRegressionTest.java
Sat Apr 20 08:22:54 2013
@@ -36,6 +36,7 @@
import java.util.Set;
import org.apache.cassandra.cql.ConnectionDetails;
+import org.apache.cassandra.thrift.ConsistencyLevel;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -47,6 +48,7 @@
private static final String KEYSPACE = testks;
private static final String TABLE = regressiontest;
//private static final String CQLV3 = 3.0.0;
+private static final String CONSISTENCY_QUORUM = QUORUM;
private static java.sql.Connection con = null;
@@ -376,6 +378,39 @@
System.out.println(resultToDisplay(result,65, with set =
empty));
+}
+
+@Test
+public void testIssue71() throws Exception
+{
+Statement stmt = con.createStatement();
+
+// Create the target Column family
+String createCF = CREATE COLUMNFAMILY t71 (k int PRIMARY KEY,
++ c text
++ ) ;;
+
+stmt.execute(createCF);
+stmt.close();
+con.close();
+
+// open it up again to see the new CF
+ con =
DriverManager.getConnection(String.format(jdbc:cassandra://%s:%d/%s?consistency=%s,HOST,PORT,KEYSPACE,CONSISTENCY_QUORUM));
+
+ // at this point defaultConsistencyLevel should be set the QUORUM
in the connection
+
+ stmt = con.createStatement();
+
+ ConsistencyLevel cl = statementExtras(stmt).getConsistencyLevel();
+ assertTrue(ConsistencyLevel.QUORUM == cl );
+
+ System.out.println();
+ System.out.println(Test Issue #71);
+ System.out.println(--);
+ System.out.println(statement.consistencyLevel = + cl);
+
+
+
}
@Test
@@ -503,12 +538,12 @@
@Test
public void isNotValid() throws Exception
{
-PreparedStatement currentStatement = ((CassandraConnection)
con).isAlive;
-PreparedStatement mockedStatement = mock(PreparedStatement.class);
-when(mockedStatement.executeQuery()).thenThrow(new SQLException(A
mocked ERROR));
-((CassandraConnection) con).isAlive = mockedStatement;
-assert con.isValid(5) == false;
-((CassandraConnection) con).isAlive = currentStatement;
+//PreparedStatement currentStatement = ((CassandraConnection)
con).isAlive;
+//PreparedStatement mockedStatement =
mock(PreparedStatement.class);
+//when(mockedStatement.executeQuery()).thenThrow(new
SQLException(A mocked ERROR));
+//((CassandraConnection) con).isAlive = mockedStatement;
+//assert con.isValid(5) == false;
+//((CassandraConnection) con).isAlive = currentStatement;
}
private final String showColumn(int index, ResultSet result) throws
SQLException
@@ -541,4 +576,11 @@
return sb.toString();
}
+
+private CassandraStatementExtras statementExtras(Statement statement)
throws Exception
+{
+Class cse =
Class.forName(org.apache.cassandra.cql.jdbc.CassandraStatementExtras);
+return (CassandraStatementExtras) statement.unwrap(cse);
+}
+
}
[jira] [Commented] (CASSANDRA-5385) IndexHelper.skipBloomFilters won't skip non-SHA filters
[
https://issues.apache.org/jira/browse/CASSANDRA-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637617#comment-13637617
]
Carl Yeksigian commented on CASSANDRA-5385:
---
{quote}
Hmm, looks like the backport blew away scrubtest on 1.2 as well. Oversight?
{quote}
Yes, was an oversight.
Just for clarification, skipBloomFilters is old.
#1 and #2 are incorrect.
1: The skipBloomFilters gets called on the index file in RowIndexEntry line 104.
2: In 2.0, the skipBloomFilters will be called on a 1.2 sstable; also, the
skipBloomFilters currently is called against any sstable when scrubbing.
IndexHelper.skipBloomFilters won't skip non-SHA filters
---
Key: CASSANDRA-5385
URL: https://issues.apache.org/jira/browse/CASSANDRA-5385
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.2.0, 2.0
Reporter: Carl Yeksigian
Assignee: Carl Yeksigian
Fix For: 1.2.4, 2.0
Attachments: 5385.patch, 5385-v2.patch
Currently, if the bloom filter is not of SHA type, we do not properly skip
the bytes. We need to read out the number of bytes, as happens in the Murmur
deserializer, then skip that many bytes instead of just skipping the hash
size. The version needs to be passed into the method as well, so that it
knows what type of index it is, and does the appropriate skipping.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5439) allow STCS options to apply to the L0 compaction performed by LCS
[ https://issues.apache.org/jira/browse/CASSANDRA-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637622#comment-13637622 ] Carl Yeksigian commented on CASSANDRA-5439: --- Changes to the original patch look good. allow STCS options to apply to the L0 compaction performed by LCS - Key: CASSANDRA-5439 URL: https://issues.apache.org/jira/browse/CASSANDRA-5439 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 2.0 Reporter: Jonathan Ellis Assignee: Carl Yeksigian Priority: Minor Fix For: 2.0 Attachments: 5439.patch, 5439-v2.txt (See CASSANDRA-5371) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5501) Missing data on SELECT on secondary index
Marco Matarazzo created CASSANDRA-5501: -- Summary: Missing data on SELECT on secondary index Key: CASSANDRA-5501 URL: https://issues.apache.org/jira/browse/CASSANDRA-5501 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Environment: linux ubuntu 12.04 Reporter: Marco Matarazzo We have a 3 nodes cluster, and a keyspace with RF = 3. From cassandra-cli everything is fine (we actually never use it, I just launched it for a check in this particular case). [default@goh_master] get agents where station_id = ascii(1110129); --- RowKey: 6c8efeb6-7209-11e2-890a-aacc0216 = (column=, value=, timestamp=1364580868176000) = (column=character_points, value=, timestamp=136103068689) = (column=component_id, value=0, timestamp=1364580868176000) = (column=corporation_id, value=3efc729e-7209-11e2-890a-aacc0216, timestamp=136103068689) = (column=entity_id, value=0, timestamp=1364580868176000) = (column=manufacturing, value=, timestamp=136103068689) = (column=model, value=55, timestamp=136103068689) = (column=name, value=Jenny Olifield, timestamp=136103068689) = (column=name_check, value=jenny_olifield, timestamp=136103068689) = (column=station_id, value=1110129, timestamp=1364580868176000) = (column=stats_intellect, value=8, timestamp=136103068689) = (column=stats_reflexes, value=8, timestamp=136103068689) = (column=stats_stamina, value=7, timestamp=136103068689) = (column=stats_technology, value=7, timestamp=136103068689) = (column=trading, value=, timestamp=136103068689) --- RowKey: dc413373-6b06-11e2-8943-aacc0216 = (column=, value=, timestamp=136656818522) = (column=character_points, value=100, timestamp=1364580381651000) = (column=component_id, value=, timestamp=1364580381651000) = (column=corporation_id, value=574934cc-6b06-11e2-a512-aacc0200, timestamp=1364580381651000) = (column=entity_id, value=0, timestamp=1364580381651000) = (column=manufacturing, value=, timestamp=1364580381651000) = (column=model, value=500018, timestamp=1364580381651000) = (column=name, value=Darren Matar, timestamp=1364580381651000) = (column=name_check, value=darren_matar, timestamp=1364580381651000) = (column=station_id, value=1110129, timestamp=1364580381651000) = (column=stats_intellect, value=10, timestamp=1364580381651000) = (column=stats_reflexes, value=10, timestamp=1364580381651000) = (column=stats_stamina, value=10, timestamp=1364580381651000) = (column=stats_technology, value=10, timestamp=1364580381651000) = (column=trading, value=1, timestamp=136656818522) --- RowKey: 0e7074ac-64bd-11e2-8c38-aacc0201 = (column=, value=, timestamp=1364828039093000) = (column=character_points, value=, timestamp=136103068676) = (column=component_id, value=0, timestamp=1364828039093000) = (column=corporation_id, value=e398294e-64bc-11e2-8c38-aacc0201, timestamp=136103068676) = (column=entity_id, value=0, timestamp=1364828039093000) = (column=manufacturing, value=1, timestamp=1362517535613000) = (column=model, value=58, timestamp=136103068676) = (column=name, value=Tom Bishop, timestamp=136103068676) = (column=name_check, value=tom_bishop, timestamp=136103068676) = (column=station_id, value=1110129, timestamp=1364828039093000) = (column=stats_intellect, value=9, timestamp=136103068676) = (column=stats_reflexes, value=7, timestamp=136103068676) = (column=stats_stamina, value=5, timestamp=136103068676) = (column=stats_technology, value=9, timestamp=136103068676) = (column=trading, value=, timestamp=136103068676) --- RowKey: 1b462f09-65f3-4148-a1a6-536b52b3bcfa = (column=, value=, timestamp=1366568185096000) = (column=character_points, value=100, timestamp=1364580381537000) = (column=component_id, value=, timestamp=1364580381537000) = (column=corporation_id, value=1d2a8803-d139-4b50-85eb-92cb1082de2e, timestamp=1364580381537000) = (column=entity_id, value=0, timestamp=1364580381537000) = (column=manufacturing, value=, timestamp=1364580381537000) = (column=model, value=53, timestamp=1364580381537000) = (column=name, value=Andrea Len, timestamp=1364580381537000) = (column=name_check, value=andrea_len, timestamp=1364580381537000) = (column=station_id, value=1110129, timestamp=1364580381537000) = (column=stats_intellect, value=10, timestamp=1364580381537000) = (column=stats_reflexes, value=10, timestamp=1364580381537000) = (column=stats_stamina, value=10, timestamp=1364580381537000) = (column=stats_technology, value=10, timestamp=1364580381537000) = (column=trading, value=1, timestamp=1366568185096000) 4 Rows Returned. From CQLSH, hovewer, the result is different, and 2 rows are missing. cqlsh:goh_master select agent_id,name,station_id from agents where station_id='1110129'; agent_id
[jira] [Created] (CASSANDRA-5502) Secondary Index Storage
Brooke Bryan created CASSANDRA-5502: --- Summary: Secondary Index Storage Key: CASSANDRA-5502 URL: https://issues.apache.org/jira/browse/CASSANDRA-5502 Project: Cassandra Issue Type: Improvement Reporter: Brooke Bryan Currently, both the CF data, and the secondary index data are stored within the same folder on disk. Being able to split these into separate folders could be a great improvement with performance. With our data, and secondary index, we will query the index a lot more, so can optimise hardware for the two traffic shapes coming through. Something like /Keyspace/CF/indexes/indexname/*.db Not too sure how much would be involved to change this, but I would imagine fairly cheap cost to reward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5503) Large Dataset with Secondary Index
Brooke Bryan created CASSANDRA-5503: --- Summary: Large Dataset with Secondary Index Key: CASSANDRA-5503 URL: https://issues.apache.org/jira/browse/CASSANDRA-5503 Project: Cassandra Issue Type: Bug Reporter: Brooke Bryan We have a cluster with 1 CF, and 1 secondary index. Currently, there are around 12 billion keys across 10 nodes, and we need to grow the cluster to support new data. (This is only a small % of our total data atm) The problem we are faced with, is when joining a new node, the system will often sit there joining, and then fail a stream stage, failing the process. This has been the result of another node running a compaction and building up its heap too high, or other issues. However, I think this problem could be massively reduced, and make the join process more stable, if the joining node pulled in all the data from the other nodes, and built up its secondary indexes after the other nodes have done everything they need to for the node to complete its join. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5385) IndexHelper.skipBloomFilters won't skip non-SHA filters
[ https://issues.apache.org/jira/browse/CASSANDRA-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637673#comment-13637673 ] Jonathan Ellis commented on CASSANDRA-5385: --- Ah, I see. So, my assertions are correct for 1.2, which is what I've been working in lately. IndexHelper.skipBloomFilters won't skip non-SHA filters --- Key: CASSANDRA-5385 URL: https://issues.apache.org/jira/browse/CASSANDRA-5385 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0, 2.0 Reporter: Carl Yeksigian Assignee: Carl Yeksigian Fix For: 1.2.4, 2.0 Attachments: 5385.patch, 5385-v2.patch Currently, if the bloom filter is not of SHA type, we do not properly skip the bytes. We need to read out the number of bytes, as happens in the Murmur deserializer, then skip that many bytes instead of just skipping the hash size. The version needs to be passed into the method as well, so that it knows what type of index it is, and does the appropriate skipping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-5385) IndexHelper.skipBloomFilters won't skip non-SHA filters
[ https://issues.apache.org/jira/browse/CASSANDRA-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637673#comment-13637673 ] Jonathan Ellis edited comment on CASSANDRA-5385 at 4/21/13 10:50 PM: - Ah, I see. So, my assertions are correct for 1.2, which is what I've been working in lately. I'll make sure it stays that way as I merge CASSANDRA-5492 forward. was (Author: jbellis): Ah, I see. So, my assertions are correct for 1.2, which is what I've been working in lately. IndexHelper.skipBloomFilters won't skip non-SHA filters --- Key: CASSANDRA-5385 URL: https://issues.apache.org/jira/browse/CASSANDRA-5385 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0, 2.0 Reporter: Carl Yeksigian Assignee: Carl Yeksigian Fix For: 1.2.4, 2.0 Attachments: 5385.patch, 5385-v2.patch Currently, if the bloom filter is not of SHA type, we do not properly skip the bytes. We need to read out the number of bytes, as happens in the Murmur deserializer, then skip that many bytes instead of just skipping the hash size. The version needs to be passed into the method as well, so that it knows what type of index it is, and does the appropriate skipping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-5385) IndexHelper.skipBloomFilters won't skip non-SHA filters
[ https://issues.apache.org/jira/browse/CASSANDRA-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637673#comment-13637673 ] Jonathan Ellis edited comment on CASSANDRA-5385 at 4/21/13 10:50 PM: - Ah, I see. So, my assertions are correct for 1.2, which is what I've been working in lately. I'll make sure it stays that way as I merge CASSANDRA-5497 and CASSANDRA-5492 forward. was (Author: jbellis): Ah, I see. So, my assertions are correct for 1.2, which is what I've been working in lately. I'll make sure it stays that way as I merge CASSANDRA-5492 forward. IndexHelper.skipBloomFilters won't skip non-SHA filters --- Key: CASSANDRA-5385 URL: https://issues.apache.org/jira/browse/CASSANDRA-5385 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.0, 2.0 Reporter: Carl Yeksigian Assignee: Carl Yeksigian Fix For: 1.2.4, 2.0 Attachments: 5385.patch, 5385-v2.patch Currently, if the bloom filter is not of SHA type, we do not properly skip the bytes. We need to read out the number of bytes, as happens in the Murmur deserializer, then skip that many bytes instead of just skipping the hash size. The version needs to be passed into the method as well, so that it knows what type of index it is, and does the appropriate skipping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5492) Backport row-level bloom filter removal to 1.2
[ https://issues.apache.org/jira/browse/CASSANDRA-5492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637674#comment-13637674 ] Jonathan Ellis commented on CASSANDRA-5492: --- Pushed a formatting fix. Backport row-level bloom filter removal to 1.2 -- Key: CASSANDRA-5492 URL: https://issues.apache.org/jira/browse/CASSANDRA-5492 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.2.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Priority: Minor Fix For: 1.2.5 With the possible presence of range tombstones, it is erroneous to skip checking for a given column in SSTableNamesIterator because the bloom filter says it is not there. This is fixed by CASSANDRA-5487, which ignores the BF, but it's a shame to leave unused bloom filters cluttering up the row cache. If it's too risky to do a full backport we can just substitute an AlwaysPresentFilter in RIE.create, but I think the code removal is fairly straightforward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: allow STCS options to apply to the L0 compaction performed by LCS patch by Carl Yeksigian; reviewed by jbellis for CASSANDRA-5439
Updated Branches:
refs/heads/trunk d11ba8ab8 - 6968f68cd
allow STCS options to apply to the L0 compaction performed by LCS
patch by Carl Yeksigian; reviewed by jbellis for CASSANDRA-5439
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6968f68c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6968f68c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6968f68c
Branch: refs/heads/trunk
Commit: 6968f68cd7c8d4a1ab4a6d4638c547e9de33a560
Parents: d11ba8a
Author: Jonathan Ellis jbel...@apache.org
Authored: Sun Apr 21 18:08:12 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Sun Apr 21 18:08:12 2013 -0500
--
CHANGES.txt|3 +-
.../db/compaction/LeveledCompactionStrategy.java |5 +-
.../cassandra/db/compaction/LeveledManifest.java | 18 ++--
.../compaction/SizeTieredCompactionStrategy.java | 65 +-
.../apache/cassandra/tools/StandaloneScrubber.java |1 +
5 files changed, 22 insertions(+), 70 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6968f68c/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2f5f5a9..9551c4c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,7 +1,8 @@
2.0
* removed PBSPredictor (CASSANDRA-5455)
* CAS support (CASSANDRA-5062, )
- * Leveled compaction performs size-tiered compactions in L0 (CASSANDRA-5371)
+ * Leveled compaction performs size-tiered compactions in L0
+ (CASSANDRA-5371, 5439)
* Add yaml network topology snitch for mixed ec2/other envs (CASSANDRA-5339)
* Log when a node is down longer than the hint window (CASSANDRA-4554)
* Optimize tombstone creation for ExpiringColumns (CASSANDRA-4917)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6968f68c/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
index f704518..0e8c2a7 100644
--- a/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
+++ b/src/java/org/apache/cassandra/db/compaction/LeveledCompactionStrategy.java
@@ -53,6 +53,7 @@ public class LeveledCompactionStrategy extends
AbstractCompactionStrategy implem
{
super(cfs, options);
int configuredMaxSSTableSize = 5;
+SizeTieredCompactionStrategyOptions localOptions = new
SizeTieredCompactionStrategyOptions(options);
if (options != null)
{
String value = options.containsKey(SSTABLE_SIZE_OPTION) ?
options.get(SSTABLE_SIZE_OPTION) : 5;
@@ -63,7 +64,7 @@ public class LeveledCompactionStrategy extends
AbstractCompactionStrategy implem
cfs.getDataTracker().subscribe(this);
logger.debug({} subscribed to the data tracker., this);
-manifest = LeveledManifest.create(cfs, this.maxSSTableSizeInMB);
+manifest = LeveledManifest.create(cfs, this.maxSSTableSizeInMB,
Collections.SSTableReaderemptyList(), localOptions);
logger.debug(Created {}, manifest);
}
@@ -347,6 +348,8 @@ public class LeveledCompactionStrategy extends
AbstractCompactionStrategy implem
uncheckedOptions.remove(SSTABLE_SIZE_OPTION);
+uncheckedOptions =
SizeTieredCompactionStrategyOptions.validateOptions(options, uncheckedOptions);
+
return uncheckedOptions;
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6968f68c/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
b/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
index 8ae0834..fb4244d 100644
--- a/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
+++ b/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
@@ -57,11 +57,13 @@ public class LeveledManifest
private final ListSSTableReader[] generations;
private final RowPosition[] lastCompactedKeys;
private final int maxSSTableSizeInBytes;
+private final SizeTieredCompactionStrategyOptions options;
-private LeveledManifest(ColumnFamilyStore cfs, int maxSSTableSizeInMB)
+private LeveledManifest(ColumnFamilyStore cfs, int maxSSTableSizeInMB,
SizeTieredCompactionStrategyOptions options)
{
this.cfs = cfs;
this.maxSSTableSizeInBytes = maxSSTableSizeInMB * 1024 * 1024;
+this.options = options;
// allocate enough generations for a PB of data
int n = (int)
git commit: add missing file
Updated Branches:
refs/heads/trunk 6968f68cd - 16806abb9
add missing file
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/16806abb
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/16806abb
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/16806abb
Branch: refs/heads/trunk
Commit: 16806abb966001c3ce358f45edcd1d316c63e7ad
Parents: 6968f68
Author: Jonathan Ellis jbel...@apache.org
Authored: Sun Apr 21 18:15:15 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Sun Apr 21 18:15:15 2013 -0500
--
.../SizeTieredCompactionStrategyOptions.java | 104 +++
1 files changed, 104 insertions(+), 0 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/16806abb/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
new file mode 100644
index 000..d7c9075
--- /dev/null
+++
b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.db.compaction;
+
+import java.util.Map;
+
+import org.apache.cassandra.exceptions.ConfigurationException;
+
+public final class SizeTieredCompactionStrategyOptions
+{
+protected static final long DEFAULT_MIN_SSTABLE_SIZE = 50L * 1024L * 1024L;
+protected static final double DEFAULT_BUCKET_LOW = 0.5;
+protected static final double DEFAULT_BUCKET_HIGH = 1.5;
+protected static final String MIN_SSTABLE_SIZE_KEY = min_sstable_size;
+protected static final String BUCKET_LOW_KEY = bucket_low;
+protected static final String BUCKET_HIGH_KEY = bucket_high;
+
+protected long minSSTableSize;
+protected double bucketLow;
+protected double bucketHigh;
+
+public SizeTieredCompactionStrategyOptions(MapString, String options)
+{
+
+String optionValue = options.get(MIN_SSTABLE_SIZE_KEY);
+minSSTableSize = optionValue == null ? DEFAULT_MIN_SSTABLE_SIZE :
Long.parseLong(optionValue);
+optionValue = options.get(BUCKET_LOW_KEY);
+bucketLow = optionValue == null ? DEFAULT_BUCKET_LOW :
Double.parseDouble(optionValue);
+optionValue = options.get(BUCKET_HIGH_KEY);
+bucketHigh = optionValue == null ? DEFAULT_BUCKET_HIGH :
Double.parseDouble(optionValue);
+}
+
+public SizeTieredCompactionStrategyOptions()
+{
+
+minSSTableSize = DEFAULT_MIN_SSTABLE_SIZE;
+bucketLow = DEFAULT_BUCKET_LOW;
+bucketHigh = DEFAULT_BUCKET_HIGH;
+}
+
+public static MapString, String validateOptions(MapString, String
options, MapString, String uncheckedOptions) throws ConfigurationException
+{
+String optionValue = options.get(MIN_SSTABLE_SIZE_KEY);
+try
+{
+long minSSTableSize = optionValue == null ?
DEFAULT_MIN_SSTABLE_SIZE : Long.parseLong(optionValue);
+if (minSSTableSize 0)
+{
+throw new ConfigurationException(String.format(%s must be non
negative: %d, MIN_SSTABLE_SIZE_KEY, minSSTableSize));
+}
+}
+catch (NumberFormatException e)
+{
+throw new ConfigurationException(String.format(%s is not a
parsable int (base10) for %s, optionValue, MIN_SSTABLE_SIZE_KEY), e);
+}
+
+double bucketLow, bucketHigh;
+optionValue = options.get(BUCKET_LOW_KEY);
+try
+{
+bucketLow = optionValue == null ? DEFAULT_BUCKET_LOW :
Double.parseDouble(optionValue);
+}
+catch (NumberFormatException e)
+{
+throw new ConfigurationException(String.format(%s is not a
parsable int (base10) for %s, optionValue, DEFAULT_BUCKET_LOW), e);
+}
+
+optionValue =
buildbot success in ASF Buildbot on cassandra-trunk
The Buildbot has detected a restored build on builder cassandra-trunk while building cassandra. Full details are available at: http://ci.apache.org/builders/cassandra-trunk/builds/2597 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch trunk] 16806abb966001c3ce358f45edcd1d316c63e7ad Blamelist: Jonathan Ellis jbel...@apache.org Build succeeded! sincerely, -The Buildbot
[jira] [Commented] (CASSANDRA-5432) Repair Freeze/Gossip Invisibility Issues 1.2.4
[ https://issues.apache.org/jira/browse/CASSANDRA-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637687#comment-13637687 ] Vijay commented on CASSANDRA-5432: -- Priam opens port for other DC's to talk to each other but nothing to do within, i still doubt the SG setup coz all IP's within a security group should be opened for both ports. May be CASSANDRA-5171 created a side effect, which i am not sure. [~jbrown] do you mind verifying it with 1.2.4? Verifying it with Priam is a bigger undertaking for me now :) Repair Freeze/Gossip Invisibility Issues 1.2.4 -- Key: CASSANDRA-5432 URL: https://issues.apache.org/jira/browse/CASSANDRA-5432 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.4 Environment: Ubuntu 10.04.1 LTS C* 1.2.3 Sun Java 6 u43 JNA Enabled Not using VNodes Reporter: Arya Goudarzi Assignee: Vijay Priority: Critical Read comment 6. This description summarizes the repair issue only, but I believe there is a bigger problem going on with networking as described on that comment. Since I have upgraded our sandbox cluster, I am unable to run repair on any node and I am reaching our gc_grace seconds this weekend. Please help. So far, I have tried the following suggestions: - nodetool scrub - offline scrub - running repair on each CF separately. Didn't matter. All got stuck the same way. The repair command just gets stuck and the machine is idling. Only the following logs are printed for repair job: INFO [Thread-42214] 2013-04-05 23:30:27,785 StorageService.java (line 2379) Starting repair command #4, repairing 1 ranges for keyspace cardspring_production INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,789 AntiEntropyService.java (line 652) [repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242] new session: will sync /X.X.X.190, /X.X.X.43, /X.X.X.56 on range (1808575600,42535295865117307932921825930779602032] for keyspace_production.[comma separated list of CFs] INFO [AntiEntropySessions:7] 2013-04-05 23:30:27,790 AntiEntropyService.java (line 858) [repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242] requesting merkle trees for BusinessConnectionIndicesEntries (to [/X.X.X.43, /X.X.X.56, /X.X.X.190]) INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,086 AntiEntropyService.java (line 214) [repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242] Received merkle tree for ColumnFamilyName from /X.X.X.43 INFO [AntiEntropyStage:1] 2013-04-05 23:30:28,147 AntiEntropyService.java (line 214) [repair #cc5a9aa0-9e48-11e2-98ba-11bde7670242] Received merkle tree for ColumnFamilyName from /X.X.X.56 Please advise. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5417) Push composites support in the storage engine
[
https://issues.apache.org/jira/browse/CASSANDRA-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637737#comment-13637737
]
T Jake Luciani commented on CASSANDRA-5417:
---
I've posted a couple minor commits to
https://github.com/tjake/cassandra/tree/5417
There was one failing test and one performance regression.
I ran some benchmarks to gauge the effectiveness of this change.
I created a COMPACT table with a few wide rows and created 1k sstables. I
benchmarked the time it took to perform a major compaction (since that causes
the most Composite comparisons).
First the base
{code}
INFO 20:54:41,504 Compacted 942 sstables to
[/var/lib/cassandra/data/mjff/data/mjff-data-ja-963,]. 1,004,572,909 bytes to
999,124,671 (~99% of original) in 372,638ms = 2.557011MB/s. 947 total rows, 6
unique.
{code}
With this change.
{code}
INFO 22:25:46,747 Compacted 942 sstables to
[/var/lib/cassandra/data/mjff/data/mjff-data-ja-963,]. 1,004,572,909 bytes to
999,124,671 (~99% of original) in 341,939ms = 2.786578MB/s. 947 total rows, 6
unique.
{code}
So a bit of a improvement (~10%).
Then I tested the performance of reading as quickly as possible. This
unfortunately was slower (~30%)
I've attached a screenshot of the current bottleneck if you want to take a look.
Push composites support in the storage engine
-
Key: CASSANDRA-5417
URL: https://issues.apache.org/jira/browse/CASSANDRA-5417
Project: Cassandra
Issue Type: Improvement
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Fix For: 2.0
CompositeType happens to be very useful and is now widely used: CQL3 heavily
rely on it, and super columns are now using it too internally. Besides,
CompositeType has been advised as a replacement of super columns on the
thrift side for a while, so it's safe to assume that it's generally used
there too.
CompositeType has initially been introduced as just another AbstractType.
Meaning that the storage engine has no nothing whatsoever of composites
being, well, composite. This has the following drawbacks:
* Because internally a composite value is handled as just a ByteBuffer, we
end up doing a lot of extra work. Typically, each time we compare 2 composite
value, we end up deserializing the components (which, while it doesn't copy
data per-se because we just slice the global ByteBuffer, still waste some cpu
cycles and allocate a bunch of ByteBuffer objects). And since compare can be
called *a lot*, this is likely not negligible.
* This make CQL3 code uglier than necessary. Basically, CQL3 makes extensive
use of composites, and since it gets backs ByteBuffer from the internal
columns, it always have to check if it's actually a compositeType or not, and
then split it and pick the different parts it needs. It's only an API
problem, but having things exposed as composites directly would definitively
make thinks cleaner. In particular, in most cases, CQL3 don't care whether it
has a composite with only one component or a non-really-composite value, but
we still always distinguishes both cases. Lastly, if we do expose composites
more directly internally, it's not a lot more work to internalize better
the different parts of the cell name that CQL3 uses (what's the clustering
key, what's the actuall CQL3 column name, what's the collection element),
making things cleaner. Last but not least, there is currently a bunch of
places where methods take a ByteBuffer as argument and it's hard to know
whether it expects a cell name or a CQL3 column name. This is pretty error
prone.
* It makes it hard (or impossible) to do a number of performance
improvements. Consider CASSANDRA-4175, I'm not really sure how you can do it
properly (in memory) if cell names are just ByteBuffer (since CQL3 column
names are just one of the component in general). But we also miss
oportunities of sharing prefixes. If we were able to share prefixes of
composite names in memory we would 1) lower the memory footprint and 2)
potentially speed-up comparison (of the prefixes) by checking reference
equality first (also, doing prefix sharing on-disk, which is a separate
concern btw, might be easier to do if we do prefix sharing in memory).
So I suggest pushing CompositeType support inside the storage engine. What I
mean by that concretely would be change the internal {{Column.name}} from
ByteBuffer to some CellName type. A CellName would API-wise just be a list of
ByteBuffer. But in practice, we'd have a specific CellName implementation for
not-really-composite names, and the truly composite implementation will allow
some prefix sharing. From an external API however, nothing would change, we
would pack the composite as usual
[jira] [Comment Edited] (CASSANDRA-5417) Push composites support in the storage engine
[
https://issues.apache.org/jira/browse/CASSANDRA-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637737#comment-13637737
]
T Jake Luciani edited comment on CASSANDRA-5417 at 4/22/13 2:45 AM:
I've posted a couple minor commits to
https://github.com/tjake/cassandra/tree/5417
There was one failing test and one performance regression.
I ran some benchmarks to gauge the effectiveness of this change.
I created a COMPACT table with a few wide rows and created 1k sstables. I
benchmarked the time it took to perform a major compaction (since that causes
the most Composite comparisons).
First the base
{code}
INFO 20:54:41,504 Compacted 942 sstables to
[/var/lib/cassandra/data/mjff/data/mjff-data-ja-963,]. 1,004,572,909 bytes to
999,124,671 (~99% of original) in 372,638ms = 2.557011MB/s. 947 total rows, 6
unique.
{code}
With this change.
{code}
INFO 22:25:46,747 Compacted 942 sstables to
[/var/lib/cassandra/data/mjff/data/mjff-data-ja-963,]. 1,004,572,909 bytes to
999,124,671 (~99% of original) in 341,939ms = 2.786578MB/s. 947 total rows, 6
unique.
{code}
So a bit of a improvement (~10%).
Then I tested the performance of reading as quickly as possible. This
unfortunately was slower (~30%)
I've attached a screenshot of the current bottleneck if you want to take a look.
http://i.imgur.com/rrp3fsX.png
was (Author: tjake):
I've posted a couple minor commits to
https://github.com/tjake/cassandra/tree/5417
There was one failing test and one performance regression.
I ran some benchmarks to gauge the effectiveness of this change.
I created a COMPACT table with a few wide rows and created 1k sstables. I
benchmarked the time it took to perform a major compaction (since that causes
the most Composite comparisons).
First the base
{code}
INFO 20:54:41,504 Compacted 942 sstables to
[/var/lib/cassandra/data/mjff/data/mjff-data-ja-963,]. 1,004,572,909 bytes to
999,124,671 (~99% of original) in 372,638ms = 2.557011MB/s. 947 total rows, 6
unique.
{code}
With this change.
{code}
INFO 22:25:46,747 Compacted 942 sstables to
[/var/lib/cassandra/data/mjff/data/mjff-data-ja-963,]. 1,004,572,909 bytes to
999,124,671 (~99% of original) in 341,939ms = 2.786578MB/s. 947 total rows, 6
unique.
{code}
So a bit of a improvement (~10%).
Then I tested the performance of reading as quickly as possible. This
unfortunately was slower (~30%)
I've attached a screenshot of the current bottleneck if you want to take a look.
Push composites support in the storage engine
-
Key: CASSANDRA-5417
URL: https://issues.apache.org/jira/browse/CASSANDRA-5417
Project: Cassandra
Issue Type: Improvement
Reporter: Sylvain Lebresne
Assignee: Sylvain Lebresne
Fix For: 2.0
CompositeType happens to be very useful and is now widely used: CQL3 heavily
rely on it, and super columns are now using it too internally. Besides,
CompositeType has been advised as a replacement of super columns on the
thrift side for a while, so it's safe to assume that it's generally used
there too.
CompositeType has initially been introduced as just another AbstractType.
Meaning that the storage engine has no nothing whatsoever of composites
being, well, composite. This has the following drawbacks:
* Because internally a composite value is handled as just a ByteBuffer, we
end up doing a lot of extra work. Typically, each time we compare 2 composite
value, we end up deserializing the components (which, while it doesn't copy
data per-se because we just slice the global ByteBuffer, still waste some cpu
cycles and allocate a bunch of ByteBuffer objects). And since compare can be
called *a lot*, this is likely not negligible.
* This make CQL3 code uglier than necessary. Basically, CQL3 makes extensive
use of composites, and since it gets backs ByteBuffer from the internal
columns, it always have to check if it's actually a compositeType or not, and
then split it and pick the different parts it needs. It's only an API
problem, but having things exposed as composites directly would definitively
make thinks cleaner. In particular, in most cases, CQL3 don't care whether it
has a composite with only one component or a non-really-composite value, but
we still always distinguishes both cases. Lastly, if we do expose composites
more directly internally, it's not a lot more work to internalize better
the different parts of the cell name that CQL3 uses (what's the clustering
key, what's the actuall CQL3 column name, what's the collection element),
making things cleaner. Last but not least, there is currently a bunch of
places where methods take a ByteBuffer as argument and it's hard to know
whether
git commit: use long math for long values
Updated Branches:
refs/heads/cassandra-1.1 7d36c1e02 - 8bb5487ea
use long math for long values
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8bb5487e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8bb5487e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8bb5487e
Branch: refs/heads/cassandra-1.1
Commit: 8bb5487eab3d91398c6a8f1f3d78b6a41aae697d
Parents: 7d36c1e
Author: Dave Brosius dbros...@apache.org
Authored: Sun Apr 21 22:53:49 2013 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Sun Apr 21 22:53:49 2013 -0400
--
.../org/apache/cassandra/utils/SlabAllocator.java |4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8bb5487e/src/java/org/apache/cassandra/utils/SlabAllocator.java
--
diff --git a/src/java/org/apache/cassandra/utils/SlabAllocator.java
b/src/java/org/apache/cassandra/utils/SlabAllocator.java
index 24a2f65..4c6c0b6 100644
--- a/src/java/org/apache/cassandra/utils/SlabAllocator.java
+++ b/src/java/org/apache/cassandra/utils/SlabAllocator.java
@@ -115,7 +115,7 @@ public class SlabAllocator extends Allocator
*/
public long getMinimumSize()
{
-return unslabbed.get() + (regionCount - 1) * REGION_SIZE;
+return unslabbed.get() + (regionCount - 1) * (long)REGION_SIZE;
}
/**
@@ -123,7 +123,7 @@ public class SlabAllocator extends Allocator
*/
public long getMaximumSize()
{
-return unslabbed.get() + regionCount * REGION_SIZE;
+return unslabbed.get() + regionCount * (long)REGION_SIZE;
}
/**
[1/2] git commit: use long math for long values
Updated Branches:
refs/heads/cassandra-1.2 d5c0c7ae3 - c5f4cdd2b
use long math for long values
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8bb5487e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8bb5487e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8bb5487e
Branch: refs/heads/cassandra-1.2
Commit: 8bb5487eab3d91398c6a8f1f3d78b6a41aae697d
Parents: 7d36c1e
Author: Dave Brosius dbros...@apache.org
Authored: Sun Apr 21 22:53:49 2013 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Sun Apr 21 22:53:49 2013 -0400
--
.../org/apache/cassandra/utils/SlabAllocator.java |4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8bb5487e/src/java/org/apache/cassandra/utils/SlabAllocator.java
--
diff --git a/src/java/org/apache/cassandra/utils/SlabAllocator.java
b/src/java/org/apache/cassandra/utils/SlabAllocator.java
index 24a2f65..4c6c0b6 100644
--- a/src/java/org/apache/cassandra/utils/SlabAllocator.java
+++ b/src/java/org/apache/cassandra/utils/SlabAllocator.java
@@ -115,7 +115,7 @@ public class SlabAllocator extends Allocator
*/
public long getMinimumSize()
{
-return unslabbed.get() + (regionCount - 1) * REGION_SIZE;
+return unslabbed.get() + (regionCount - 1) * (long)REGION_SIZE;
}
/**
@@ -123,7 +123,7 @@ public class SlabAllocator extends Allocator
*/
public long getMaximumSize()
{
-return unslabbed.get() + regionCount * REGION_SIZE;
+return unslabbed.get() + regionCount * (long)REGION_SIZE;
}
/**
[2/2] git commit: Merge branch 'cassandra-1.1' into cassandra-1.2
Merge branch 'cassandra-1.1' into cassandra-1.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c5f4cdd2 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c5f4cdd2 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c5f4cdd2 Branch: refs/heads/cassandra-1.2 Commit: c5f4cdd2be439dc3057bddcc6cbdd7ded0938ae3 Parents: d5c0c7a 8bb5487 Author: Dave Brosius dbros...@apache.org Authored: Sun Apr 21 22:55:04 2013 -0400 Committer: Dave Brosius dbros...@apache.org Committed: Sun Apr 21 22:55:04 2013 -0400 -- .../org/apache/cassandra/utils/SlabAllocator.java |4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/c5f4cdd2/src/java/org/apache/cassandra/utils/SlabAllocator.java --
