[jira] [Updated] (CASSANDRA-5443) Add CAS CQL support
[ https://issues.apache.org/jira/browse/CASSANDRA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-5443: Attachment: 0006-Make-EXISTS-a-reserverd-keyword.txt Add CAS CQL support --- Key: CASSANDRA-5443 URL: https://issues.apache.org/jira/browse/CASSANDRA-5443 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Jonathan Ellis Assignee: Sylvain Lebresne Fix For: 2.0 Attachments: 0001-Refactor-Update-and-Delete-statement-to-extract-common.txt, 0002-Add-syntax-to-support-conditional-update-delete.txt, 0003-Handle-deleted-and-expiring-column-in-paxos-updates.txt, 0004-Support-tombstones-when-comparing-for-CAS.txt, 0005-add-UPDATE-.-IF-NOT-EXISTS-syntax.txt, 0006-Make-EXISTS-a-reserverd-keyword.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5443) Add CAS CQL support
[
https://issues.apache.org/jira/browse/CASSANDRA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645346#comment-13645346
]
Sylvain Lebresne commented on CASSANDRA-5443:
-
bq. K_EXISTS has not been added to unreserved keywords
Right, good catch, adding a patch to make that happen. I'm also not sure about
{{IF}} and {{NOT}}, it will weird to have them unreserved and I can't imagine
someone in his right mind using them as column name (I note that for the same
reasoning, {{NULL}} has been added as a reserved keyword recently).
Add CAS CQL support
---
Key: CASSANDRA-5443
URL: https://issues.apache.org/jira/browse/CASSANDRA-5443
Project: Cassandra
Issue Type: Sub-task
Components: API, Core
Reporter: Jonathan Ellis
Assignee: Sylvain Lebresne
Fix For: 2.0
Attachments:
0001-Refactor-Update-and-Delete-statement-to-extract-common.txt,
0002-Add-syntax-to-support-conditional-update-delete.txt,
0003-Handle-deleted-and-expiring-column-in-paxos-updates.txt,
0004-Support-tombstones-when-comparing-for-CAS.txt,
0005-add-UPDATE-.-IF-NOT-EXISTS-syntax.txt,
0006-Make-EXISTS-a-reserverd-keyword.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5521) move IndexSummary off heap
[ https://issues.apache.org/jira/browse/CASSANDRA-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645365#comment-13645365 ] Vijay commented on CASSANDRA-5521: -- Pushed an update to https://github.com/Vijay2win/cassandra/commits/5521_v2 This update doesn't use int[] which v1 used, v2 uses kind of offheap index over index summary which is stored at the beginning of the memory block... Moves index summary completely offheap... :) I was not able to see any difference in performance between trunk and v2 using stress tool. Micro benchmark shows v2 is 6 seconds slower for 20 Bilion get's hence it is was not noticeable in stress tool, but the real benefit is less pauses and more data a node can hold... move IndexSummary off heap -- Key: CASSANDRA-5521 URL: https://issues.apache.org/jira/browse/CASSANDRA-5521 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jonathan Ellis Assignee: Vijay Fix For: 2.0 IndexSummary can still use a lot of heap for narrow-row sstables. (It can also contribute to memory fragmentation because of the large arrays it creates.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5525) Adding nodes to 1.2 cluster w/ vnodes streamed more data than average node load
[
https://issues.apache.org/jira/browse/CASSANDRA-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645445#comment-13645445
]
Richard Low commented on CASSANDRA-5525:
Could you attach the output of 'nodetool ring' to list all the tokens? Also
what is your replication factor?
There is a balancing problem when adding new nodes without running shuffle (or
decommissioning and bootstrapping each node). When Cassandra increases the
number of tokens from 1 to N (256 in your case), it splits the original ranges
into N consecutive ranges. This doesn't change where the data lives but does
increase the number of tokens.
Cassandra knows that the adjacent tokens are on the same node so doesn't try to
store replicas on the same node. It looks for the next range on another node,
just like how multi DC replication ensure replicas are in different data
centers.
Now when a new node is added, it doesn't choose adjacent tokens, it has them
spread randomly around the ring. Just one of these small ranges could hold
replicas for lots of data, because it becomes the next node in the ring. For
high enough replication factor and certain (quite likely) choices of tokens, a
new node could end up storing 100% of the data. This could explain what you
are seeing, will need to see the token list and RF to confirm.
Adding nodes to 1.2 cluster w/ vnodes streamed more data than average node
load
---
Key: CASSANDRA-5525
URL: https://issues.apache.org/jira/browse/CASSANDRA-5525
Project: Cassandra
Issue Type: Bug
Reporter: John Watson
Attachments: Screen Shot 2013-04-25 at 12.35.24 PM.png
12 node cluster upgraded from 1.1.9 to 1.2.3, enabled 'num_tokens: 256',
restarted and ran upgradesstables and cleanup.
Tried to join 2 additional nodes into the ring.
However, 1 of the new nodes ran out of disk space. This started causing 'no
host id' alerts in the live cluster when attempting to store hints for that
node.
{noformat}
ERROR 10:12:02,408 Exception in thread Thread[MutationStage:190,5,main]
java.lang.AssertionError: Missing host ID
{noformat}
The other node I killed to stop it from continuing to join. Since the live
cluster was now in some sort of broken state dropping mutation messages on 3
nodes. This was fixed by restarting them, however 1 node never stopped, so
had to decomm it (leaving the original cluster at 11 nodes.)
Ring pre-join:
{noformat}
Load Tokens Owns (effective) Host ID
147.55 GB 256 16.7% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27
124.99 GB 256 16.7% 93f4400a-09d9-4ca0-b6a6-9bcca2427450
136.63 GB 256 16.7% ff821e8e-b2ca-48a9-ac3f-8234b16329ce
141.78 GB 253 100.0%339c474f-cf19-4ada-9a47-8b10912d5eb3
137.74 GB 256 16.7% 6d726cbf-147d-426e-a735-e14928c95e45
135.9 GB 256 16.7% e59a02b3-8b91-4abd-990e-b3cb2a494950
165.96 GB 256 16.7% 83ca527c-60c5-4ea0-89a8-de53b92b99c8
135.41 GB 256 16.7% c3ea4026-551b-4a14-a346-480e8c1fe283
143.38 GB 256 16.7% df7ba879-74ad-400b-b371-91b45dcbed37
178.05 GB 256 25.0% 78192d73-be0b-4d49-a129-9bec0770efed
194.92 GB 256 25.0% 361d7e31-b155-4ce1-8890-451b3ddf46cf
150.5 GB 256 16.7% 9889280a-1433-439e-bb84-6b7e7f44d761
{noformat}
Ring after decomm bad node:
{noformat}
Load Tokens Owns (effective) Host ID
80.95 GB 256 16.7% 754f9f4c-4ba7-4495-97e7-1f5b6755cb27
87.15 GB 256 16.7% 93f4400a-09d9-4ca0-b6a6-9bcca2427450
98.16 GB 256 16.7% ff821e8e-b2ca-48a9-ac3f-8234b16329ce
142.6 GB 253 100.0%339c474f-cf19-4ada-9a47-8b10912d5eb3
77.64 GB 256 16.7% e59a02b3-8b91-4abd-990e-b3cb2a494950
194.31 GB 256 25.0% 6d726cbf-147d-426e-a735-e14928c95e45
221.94 GB 256 33.3% 83ca527c-60c5-4ea0-89a8-de53b92b99c8
87.61 GB 256 16.7% c3ea4026-551b-4a14-a346-480e8c1fe283
101.02 GB 256 16.7% df7ba879-74ad-400b-b371-91b45dcbed37
172.44 GB 256 25.0% 78192d73-be0b-4d49-a129-9bec0770efed
108.5 GB 256 16.7% 9889280a-1433-439e-bb84-6b7e7f44d761
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5443) Add CAS CQL support
[ https://issues.apache.org/jira/browse/CASSANDRA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645516#comment-13645516 ] Aleksey Yeschenko commented on CASSANDRA-5443: -- +1 Add CAS CQL support --- Key: CASSANDRA-5443 URL: https://issues.apache.org/jira/browse/CASSANDRA-5443 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Jonathan Ellis Assignee: Sylvain Lebresne Fix For: 2.0 Attachments: 0001-Refactor-Update-and-Delete-statement-to-extract-common.txt, 0002-Add-syntax-to-support-conditional-update-delete.txt, 0003-Handle-deleted-and-expiring-column-in-paxos-updates.txt, 0004-Support-tombstones-when-comparing-for-CAS.txt, 0005-add-UPDATE-.-IF-NOT-EXISTS-syntax.txt, 0006-Make-EXISTS-a-reserverd-keyword.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-5514) Allow timestamp hints
[ https://issues.apache.org/jira/browse/CASSANDRA-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645010#comment-13645010 ] Tupshin Harper edited comment on CASSANDRA-5514 at 4/30/13 12:06 PM: - See CASSANDRA-3581 for a previous ticket along similar lines. +1 on no additional syntax and Sylvain's general approach. +1000 on a time-series aware compaction strategy. That is where the big win will be. was (Author: tupshin): See CASSANDRA-5518 for a previous ticket along similar lines. +1 on no additional syntax and Sylvain's general approach. +1000 on a time-series aware compaction strategy. That is where the big win will be. Allow timestamp hints - Key: CASSANDRA-5514 URL: https://issues.apache.org/jira/browse/CASSANDRA-5514 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Assignee: Marcus Eriksson Fix For: 2.0 Slice queries can't optimize based on timestamp except for rare cases (CASSANDRA-4116). However, many common queries involve an implicit time component, where the application author knows that he is only interested in data more recent than X, or older than Y. We could use the per-sstable max and min timestamps we track to avoid touching cold data if we could pass a hint to Cassandra about the time range we care about. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5514) Allow timestamp hints
[ https://issues.apache.org/jira/browse/CASSANDRA-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645519#comment-13645519 ] Tupshin Harper commented on CASSANDRA-5514: --- Markus, fixed to 3581. Allow timestamp hints - Key: CASSANDRA-5514 URL: https://issues.apache.org/jira/browse/CASSANDRA-5514 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Assignee: Marcus Eriksson Fix For: 2.0 Slice queries can't optimize based on timestamp except for rare cases (CASSANDRA-4116). However, many common queries involve an implicit time component, where the application author knows that he is only interested in data more recent than X, or older than Y. We could use the per-sstable max and min timestamps we track to avoid touching cold data if we could pass a hint to Cassandra about the time range we care about. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-5443) Add CAS CQL support
[ https://issues.apache.org/jira/browse/CASSANDRA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne resolved CASSANDRA-5443. - Resolution: Fixed Committed, thanks Add CAS CQL support --- Key: CASSANDRA-5443 URL: https://issues.apache.org/jira/browse/CASSANDRA-5443 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Jonathan Ellis Assignee: Sylvain Lebresne Fix For: 2.0 Attachments: 0001-Refactor-Update-and-Delete-statement-to-extract-common.txt, 0002-Add-syntax-to-support-conditional-update-delete.txt, 0003-Handle-deleted-and-expiring-column-in-paxos-updates.txt, 0004-Support-tombstones-when-comparing-for-CAS.txt, 0005-add-UPDATE-.-IF-NOT-EXISTS-syntax.txt, 0006-Make-EXISTS-a-reserverd-keyword.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: Fix counter updates broken by 5443 patch
Updated Branches:
refs/heads/trunk e431fb722 - 2f3f620ed
Fix counter updates broken by 5443 patch
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2f3f620e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2f3f620e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2f3f620e
Branch: refs/heads/trunk
Commit: 2f3f620ed1bdd1bbcf5564b75885b9aaf02c0359
Parents: e431fb7
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Tue Apr 30 14:46:37 2013 +0200
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Apr 30 14:46:37 2013 +0200
--
.../cql3/statements/ModificationStatement.java | 84 ---
1 files changed, 43 insertions(+), 41 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2f3f620e/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
index 542b6d6..7ca 100644
--- a/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/ModificationStatement.java
@@ -476,54 +476,56 @@ public abstract class ModificationStatement implements
CQLStatement
ModificationStatement stmt = prepareInternal(cfDef, boundNames);
-if (stmt.isCounter())
-throw new InvalidRequestException(Conditional updates are not
supported on counter tables);
-
-if (ifNotExists)
-{
-// To have both 'IF NOT EXISTS' and some other conditions
doesn't make sense.
-// So far this is enforced by the parser, but let's assert it
for sanity if ever the parse changes.
-assert conditions.isEmpty();
-stmt.setIfNotExistCondition();
-}
-else
+if (stmt.hasConditions())
{
-for (PairColumnIdentifier, Operation.RawUpdate entry :
conditions)
-{
-CFDefinition.Name name = cfDef.get(entry.left);
-if (name == null)
-throw new
InvalidRequestException(String.format(Unknown identifier %s, entry.left));
-
-/*
- * Lists column names are based on a server-side generated
timeuuid. So we can't allow lists
- * operation or that would yield unexpected results
(update that should apply wouldn't). So for
- * now, we just refuse lists, which also save use from
having to bother about the read that some
- * list operation involve.
- */
-if (name.type instanceof ListType)
-throw new InvalidRequestException(String.format(List
operation (%s) are not allowed in conditional updates, name));
+if (stmt.isCounter())
+throw new InvalidRequestException(Conditional updates are
not supported on counter tables);
-Operation condition = entry.right.prepare(name);
-assert !condition.requiresRead();
+if (attrs.timestamp != null)
+throw new InvalidRequestException(Cannot provide custom
timestamp for conditional update);
-condition.collectMarkerSpecification(boundNames);
-
-switch (name.kind)
+if (ifNotExists)
+{
+// To have both 'IF NOT EXISTS' and some other conditions
doesn't make sense.
+// So far this is enforced by the parser, but let's assert
it for sanity if ever the parse changes.
+assert conditions.isEmpty();
+stmt.setIfNotExistCondition();
+}
+else
+{
+for (PairColumnIdentifier, Operation.RawUpdate entry :
conditions)
{
-case KEY_ALIAS:
-case COLUMN_ALIAS:
-throw new
InvalidRequestException(String.format(PRIMARY KEY part %s found in SET part,
entry.left));
-case VALUE_ALIAS:
-case COLUMN_METADATA:
-stmt.addCondition(condition);
-break;
+CFDefinition.Name name = cfDef.get(entry.left);
+if (name == null)
+throw new
InvalidRequestException(String.format(Unknown identifier %s, entry.left));
+
+/*
+
[jira] [Commented] (CASSANDRA-5441) Add support for read at CL.SERIAL
[ https://issues.apache.org/jira/browse/CASSANDRA-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645556#comment-13645556 ] Sylvain Lebresne commented on CASSANDRA-5441: - If we timeout trying to commit in-progress round, we should throw a timeout exception, not proceed to fetching the rows. nit: Maybe we could rewrite You must use CAS for serializable writes - You must use conditional updates for serializable writes? Add support for read at CL.SERIAL - Key: CASSANDRA-5441 URL: https://issues.apache.org/jira/browse/CASSANDRA-5441 Project: Cassandra Issue Type: Sub-task Components: API, Core Affects Versions: 2.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5514) Allow timestamp hints
[ https://issues.apache.org/jira/browse/CASSANDRA-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645563#comment-13645563 ] Jonathan Ellis commented on CASSANDRA-5514: --- I made a good point in 3581 that I forgot about. :) Currently we only promote row tombstones if there is enough data in the row to be worth indexing, under the reasoning that if it's not worth indexing we might as well read the tombstone from the data file as well as any columns involved. So, we'll need to change that to always promote (CASSANDRA-5526). As it happens, I wrote this code once already for CASSANDRA-5487, so I'll see about dusting it off. Allow timestamp hints - Key: CASSANDRA-5514 URL: https://issues.apache.org/jira/browse/CASSANDRA-5514 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Assignee: Marcus Eriksson Fix For: 2.0 Slice queries can't optimize based on timestamp except for rare cases (CASSANDRA-4116). However, many common queries involve an implicit time component, where the application author knows that he is only interested in data more recent than X, or older than Y. We could use the per-sstable max and min timestamps we track to avoid touching cold data if we could pass a hint to Cassandra about the time range we care about. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5526) Always promote row tombstones
Jonathan Ellis created CASSANDRA-5526: - Summary: Always promote row tombstones Key: CASSANDRA-5526 URL: https://issues.apache.org/jira/browse/CASSANDRA-5526 Project: Cassandra Issue Type: Sub-task Components: Core Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5150) sstable2json doesn't check SIGPIPE
[ https://issues.apache.org/jira/browse/CASSANDRA-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645627#comment-13645627 ] Pawel Mirski commented on CASSANDRA-5150: - Tested on Windows, it seems that performance drop is minimal (about 1%). sstable2json doesn't check SIGPIPE -- Key: CASSANDRA-5150 URL: https://issues.apache.org/jira/browse/CASSANDRA-5150 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 2.0 Reporter: Will Oberman Priority: Minor Labels: lhf Attachments: trunk-5150.txt I believe this explains the issue better than I can: http://stackoverflow.com/questions/11695500/how-do-i-get-java-to-exit-when-piped-to-head. Basically, I expected that if I did: sstable2json SSTABLE | other-process, and other-process had issues and/or died then the sstable2json process would die. It doesn't. My workaround is using mkfifo FILE, and having sstable2json write to FILE, other-process read from FILE, and a 3rd overall process make sure the other two processes are working. But, it would be _much_ simplier if sstable2json failed on SIGPIPE. I looks like the fix is to periodically check System.out.checkError() in the Java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5514) Allow timestamp hints
[ https://issues.apache.org/jira/browse/CASSANDRA-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645632#comment-13645632 ] Sylvain Lebresne commented on CASSANDRA-5514: - bq. So, we'll need to change that to always promote I'm not sure that's necessary for that issue. The goal here would be to ignore sstable based on basically their metadata, so promoting stuff from the data file to the index file shouldn't have any impact here. *But*, it's true that row tombstones complicate stuff because in theory, as soon as a sstable has a row tombstone, then the min/max column name should be empty (i.e. cover allthethings). But I think we can probably get away by involving the min/max timestamps. More precisely, we can ignore row tombstones in the min/max column computation and initially ignore sstable based on this min/max column name. But then, we would need to do a pass over the result to make sure no column from the result could be overridden by a row tombstone in one of the ignored sstable, which can be done using the sstable max timestamp. Or was that something else you had in mind? Allow timestamp hints - Key: CASSANDRA-5514 URL: https://issues.apache.org/jira/browse/CASSANDRA-5514 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Assignee: Marcus Eriksson Fix For: 2.0 Slice queries can't optimize based on timestamp except for rare cases (CASSANDRA-4116). However, many common queries involve an implicit time component, where the application author knows that he is only interested in data more recent than X, or older than Y. We could use the per-sstable max and min timestamps we track to avoid touching cold data if we could pass a hint to Cassandra about the time range we care about. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-5526) Always promote row tombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-5526. --- Resolution: Won't Fix Always promote row tombstones - Key: CASSANDRA-5526 URL: https://issues.apache.org/jira/browse/CASSANDRA-5526 Project: Cassandra Issue Type: Sub-task Components: Core Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5514) Allow timestamp hints
[ https://issues.apache.org/jira/browse/CASSANDRA-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645692#comment-13645692 ] Jonathan Ellis commented on CASSANDRA-5514: --- You're right, that's a simpler solution. Allow timestamp hints - Key: CASSANDRA-5514 URL: https://issues.apache.org/jira/browse/CASSANDRA-5514 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Assignee: Marcus Eriksson Fix For: 2.0 Slice queries can't optimize based on timestamp except for rare cases (CASSANDRA-4116). However, many common queries involve an implicit time component, where the application author knows that he is only interested in data more recent than X, or older than Y. We could use the per-sstable max and min timestamps we track to avoid touching cold data if we could pass a hint to Cassandra about the time range we care about. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4316) Compaction Throttle too bursty with large rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-4316: -- Attachment: 4316-1.2.txt Patch against 1.2 added. Compaction Throttle too bursty with large rows -- Key: CASSANDRA-4316 URL: https://issues.apache.org/jira/browse/CASSANDRA-4316 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.0 Reporter: Wayne Lewis Assignee: Jonathan Ellis Fix For: 1.2.5 Attachments: 4316-1.2.txt, 4316-1.2.txt, 4316-1.2-v2.txt, 4316-v3.txt In org.apache.cassandra.db.compaction.CompactionIterable the check for compaction throttling occurs once every 1000 rows. In our workload this is much too large as we have many large rows (16 - 100 MB). With a 100 MB row, about 100 GB is read (and possibly written) before the compaction throttle sleeps. This causes bursts of essentially unthrottled compaction IO followed by a long sleep which yields inconsistence performance and high error rates during the bursts. We applied a workaround to check throttle every row which solved our performance and error issues: line 116 in org.apache.cassandra.db.compaction.CompactionIterable: if ((row++ % 1000) == 0) replaced with if ((row++ % 1) == 0) I think the better solution is to calculate how often throttle should be checked based on the throttle rate to apply sleeps more consistently. E.g. if 16MB/sec is the limit then check for sleep after every 16MB is read so sleeps are spaced out about every second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5441) Add support for read at CL.SERIAL
[ https://issues.apache.org/jira/browse/CASSANDRA-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645706#comment-13645706 ] Jonathan Ellis commented on CASSANDRA-5441: --- bq. If we timeout trying to commit in-progress round, we should throw a timeout exception, not proceed to fetching the rows Not sure what you mean -- preparePaxos will throw WriteTimeoutException which will propagate all the way back to the client. https://github.com/jbellis/cassandra/commits/5441-2 rebased post-CASSANDRA-544 with nit addressed. Add support for read at CL.SERIAL - Key: CASSANDRA-5441 URL: https://issues.apache.org/jira/browse/CASSANDRA-5441 Project: Cassandra Issue Type: Sub-task Components: API, Core Affects Versions: 2.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5441) Add support for read at CL.SERIAL
[
https://issues.apache.org/jira/browse/CASSANDRA-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645727#comment-13645727
]
Sylvain Lebresne commented on CASSANDRA-5441:
-
bq. Not sure what you mean – preparePaxos will throw WriteTimeoutException
I mean, what if {{beginAndRepairPaxos}} returns {{null}} a bunch of times
(maybe we've replayed an in-progress update sucessfully but it took us some
time, and then we're unlucky and can't get our prepare 'promised' due to
contention) and we end up exiting the {{while}} loop due to the while condition
and not due to {{beginAndRepairPaxos}} returning something. Then we should
timeout, not do as if everything was fine (since it's not).
Add support for read at CL.SERIAL
-
Key: CASSANDRA-5441
URL: https://issues.apache.org/jira/browse/CASSANDRA-5441
Project: Cassandra
Issue Type: Sub-task
Components: API, Core
Affects Versions: 2.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Fix For: 2.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5441) Add support for read at CL.SERIAL
[ https://issues.apache.org/jira/browse/CASSANDRA-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645743#comment-13645743 ] Jonathan Ellis commented on CASSANDRA-5441: --- Ah, gotcha. Fix pushed. Add support for read at CL.SERIAL - Key: CASSANDRA-5441 URL: https://issues.apache.org/jira/browse/CASSANDRA-5441 Project: Cassandra Issue Type: Sub-task Components: API, Core Affects Versions: 2.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5527) Deletion by Secondary Key
Rick Branson created CASSANDRA-5527: --- Summary: Deletion by Secondary Key Key: CASSANDRA-5527 URL: https://issues.apache.org/jira/browse/CASSANDRA-5527 Project: Cassandra Issue Type: Improvement Reporter: Rick Branson Given Cassandra's popularity as a time ordered list store, the inability to do deletes by anything other than the primary key without re-implementing tombstones in the application is a bit of an achilles heel for many use cases. It's a data modeling problem that seems to come up quite often, and given that we now have the CQL3 abstraction layer sitting on top of the storage engine, I think there's an opportunity to take this burden off of the application layer. I've spent several weeks thinking about this problem within the context of Cassandra, and I think I've come up with a reasonable proposal. It would involve addition of a secondary key facility to CQL3 tables: CREATE TABLE timeline ( timeline_id uuid, entry_id timeuuid, entry_key blob, entry_payload blob, PRIMARY KEY (timeline_id, entry_id), KEY (timeline_id, entry_key) ); Secondary keys would be required to share the same partition key with the primary key. They would be included to support deletion by secondary key operations: DELETE FROM timeline WHERE timeline_id = X and entry_key = Y; Underneath, the storage engine row would contain additional secondary key tombstones. Secondary key deletion would be read-free, requiring a single tombstone write. The cost of reads would necessarily go up. Queries would need to be modified to perform an additional step to find any matching secondary key tombstones and perform the regular convergence process. The secondary key tombstones should be cleaned up by the regular tombstone GC process. While I didn't want to complicate this idea too much, it might be also worth having a discussion around supporting secondary key queries as well, or at least making the schema compatible with potential future support (maybe rename KEY to DELETABLE KEY or something). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5509) Decouple Consistency Durability
[ https://issues.apache.org/jira/browse/CASSANDRA-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645755#comment-13645755 ] Jonathan Ellis commented on CASSANDRA-5509: --- Yup, that's right. That's where the more robust code to reject writes with UAE if hints are getting backed up comes in. But, tracking hint backlog locally is not quite the same as doing it for the cluster. Definitely possible, just something to be cautious about, since I don't think a regression here would be easy to spot in dtests. Decouple Consistency Durability - Key: CASSANDRA-5509 URL: https://issues.apache.org/jira/browse/CASSANDRA-5509 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Rick Branson Right now in Cassandra, consistency and durability are intertwined in a way that is unnecessary. In environments where nodes have unreliable local storage, the consistency level of writes must be increased to N+1 to ensure that N host failure(s) don't cause data loss, even if it's acceptable that consistency is weaker. The probability of data loss is also heavily influenced by entropy. An example is if the client chooses a replica as the write coordinator for a CL.ONE write, the risk of losing that data increases substantially. During a node outage, the chance of data loss is elevated for a relatively long time: the entire length of the node outage + recovery time. The required increase in consistency level has real impact: it creates the potential for availability issues during routine maintenance as an unlucky node failure can cause writes to start failing. It's also generally considered a best practice that each datacenter has at least 3 replicas of data, even if quorums for consistency are not required, as it's the only way to ensure strong durability in the face of transient inter-DC failures. I found a relevant paper that provides some theoretical grounding while researching: http://www.cs.ubc.ca/~geoffrey/papers/durability-sigops04.pdf I'd like to propose that in the event of a down replica, the coordinator attempts to achieve RF by distributing remote hints to RF-liveReplicaCount non-replica nodes. If the coordinator itself is a non-replica, it would be an acceptable choice for a remote hint as well. This would achieve RF level durability without the availability penalty of increasing consistency. This would also allow decreasing the (global) RF, as RF durability goals could still be achieved during transient inter-DC failures, requiring just RF nodes in each DC, instead of RF replicas in each DC. Even better would be if the selection of remote hint nodes respected the replication strategy and was able to achieve the cross-rack / cross-DC durability. While ConsistencyLevel is a pretty overloaded concept at this point, and I think it'd be great to add a DurabilityLevel to each write, I understand that this is likely not pragmatic. Therefore, considering that the CL.TWO and CL.THREE options were added largely for durability reasons, I propose that they be repurposed to support durability goals and remote hinting. They would require 1 replica ACK and CL-1 (replica|hint) ACKs. It also might be desirable to extend the ANY option to require multiple hint ACKs, such as CL.ANY_TWO or CL.ANY_THREE, which would support combined very high durability and very high availability. All CLs will benefit as remote hinting vastly tightens the window of elevated data loss chance during a node outage from nodeOutageDuration + recoveryDuration to the time it takes for the coordinator to distribute remote hints. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5527) Deletion by Secondary Key
[
https://issues.apache.org/jira/browse/CASSANDRA-5527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645764#comment-13645764
]
Jonathan Ellis commented on CASSANDRA-5527:
---
bq. Underneath, the storage engine row would contain additional secondary key
tombstones
Hmm, I think you might be glossing over something problematic here.
Currently we support three types of tombstones:
- Partition key tombstones, which are just a an int and a long (local and
client-facing deletion times)
- Range tombstones, which are an int/long pair with a start and stop cell name
(in the conveniently named {{RangeTombstone}} class)
- Single-cell tombstones
Partition key tombstones are just hardcoded to come after the PK itself in the
row header. Range tombstones are scattered among the data cells, following the
same comparator rules. So if we are looking for cell X, the same scan we'd do
for X will also run across anything tombstoning it without having to do extra
seeks. (We'll replicate a range tombstone multiple times if it covers multiple
cell-name-index blocks.)
The problem is that I don't see a way to efficiently check for tombstones
against cell names that are not part of the PK (hence, comparator). If we're
talking about loading a list of key tombstones from the row header and
checking each one in turn then I think I'm -1 on the idea.
Deletion by Secondary Key
-
Key: CASSANDRA-5527
URL: https://issues.apache.org/jira/browse/CASSANDRA-5527
Project: Cassandra
Issue Type: Improvement
Reporter: Rick Branson
Given Cassandra's popularity as a time ordered list store, the inability to
do deletes by anything other than the primary key without re-implementing
tombstones in the application is a bit of an achilles heel for many use
cases. It's a data modeling problem that seems to come up quite often, and
given that we now have the CQL3 abstraction layer sitting on top of the
storage engine, I think there's an opportunity to take this burden off of the
application layer. I've spent several weeks thinking about this problem
within the context of Cassandra, and I think I've come up with a reasonable
proposal.
It would involve addition of a secondary key facility to CQL3 tables:
CREATE TABLE timeline (
timeline_id uuid,
entry_id timeuuid,
entry_key blob,
entry_payload blob,
PRIMARY KEY (timeline_id, entry_id),
KEY (timeline_id, entry_key)
);
Secondary keys would be required to share the same partition key with the
primary key. They would be included to support deletion by secondary key
operations:
DELETE FROM timeline WHERE timeline_id = X and entry_key = Y;
Underneath, the storage engine row would contain additional secondary key
tombstones. Secondary key deletion would be read-free, requiring a single
tombstone write. The cost of reads would necessarily go up. Queries would
need to be modified to perform an additional step to find any matching
secondary key tombstones and perform the regular convergence process. The
secondary key tombstones should be cleaned up by the regular tombstone GC
process.
While I didn't want to complicate this idea too much, it might be also worth
having a discussion around supporting secondary key queries as well, or at
least making the schema compatible with potential future support (maybe
rename KEY to DELETABLE KEY or something).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5441) Add support for read at CL.SERIAL
[ https://issues.apache.org/jira/browse/CASSANDRA-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645765#comment-13645765 ] Sylvain Lebresne commented on CASSANDRA-5441: - +1 Add support for read at CL.SERIAL - Key: CASSANDRA-5441 URL: https://issues.apache.org/jira/browse/CASSANDRA-5441 Project: Cassandra Issue Type: Sub-task Components: API, Core Affects Versions: 2.0 Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: add read support for CL.SERIAL patch by jbellis; reviewed by slebresne for CASSANDRA-5441
Updated Branches:
refs/heads/trunk 2f3f620ed - 122775166
add read support for CL.SERIAL
patch by jbellis; reviewed by slebresne for CASSANDRA-5441
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/12277516
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/12277516
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/12277516
Branch: refs/heads/trunk
Commit: 1227751661e653fb39be1ada369a9eda8f1e9d7f
Parents: 2f3f620
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Apr 30 12:42:26 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Apr 30 12:42:26 2013 -0500
--
CHANGES.txt|2 +-
interface/cassandra.thrift |1 +
.../apache/cassandra/thrift/ConsistencyLevel.java |5 +-
.../org/apache/cassandra/cql/QueryProcessor.java |2 +-
.../org/apache/cassandra/db/ConsistencyLevel.java | 10 +-
.../org/apache/cassandra/service/StorageProxy.java | 162 ++-
.../org/apache/cassandra/service/paxos/Commit.java |2 +-
.../apache/cassandra/thrift/ThriftConversion.java | 17 +--
test/system/test_thrift_server.py |3 +
9 files changed, 127 insertions(+), 77 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/12277516/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 8f2f3c0..5e9edfd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -2,7 +2,7 @@
* Removed compatibility with pre-1.2.5 sstables and network messages
(CASSANDRA-5511)
* removed PBSPredictor (CASSANDRA-5455)
- * CAS support (CASSANDRA-5062, )
+ * CAS support (CASSANDRA-5062, 5441, 5443)
* Leveled compaction performs size-tiered compactions in L0
(CASSANDRA-5371, 5439)
* Add yaml network topology snitch for mixed ec2/other envs (CASSANDRA-5339)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/12277516/interface/cassandra.thrift
--
diff --git a/interface/cassandra.thrift b/interface/cassandra.thrift
index a57603c..b057fa0 100644
--- a/interface/cassandra.thrift
+++ b/interface/cassandra.thrift
@@ -228,6 +228,7 @@ enum ConsistencyLevel {
ANY = 6,
TWO = 7,
THREE = 8,
+SERIAL = 9,
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/12277516/interface/thrift/gen-java/org/apache/cassandra/thrift/ConsistencyLevel.java
--
diff --git
a/interface/thrift/gen-java/org/apache/cassandra/thrift/ConsistencyLevel.java
b/interface/thrift/gen-java/org/apache/cassandra/thrift/ConsistencyLevel.java
index 8386797..96b6308 100644
---
a/interface/thrift/gen-java/org/apache/cassandra/thrift/ConsistencyLevel.java
+++
b/interface/thrift/gen-java/org/apache/cassandra/thrift/ConsistencyLevel.java
@@ -82,7 +82,8 @@ public enum ConsistencyLevel implements
org.apache.thrift.TEnum {
ALL(5),
ANY(6),
TWO(7),
- THREE(8);
+ THREE(8),
+ SERIAL(9);
private final int value;
@@ -119,6 +120,8 @@ public enum ConsistencyLevel implements
org.apache.thrift.TEnum {
return TWO;
case 8:
return THREE;
+ case 9:
+return SERIAL;
default:
return null;
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/12277516/src/java/org/apache/cassandra/cql/QueryProcessor.java
--
diff --git a/src/java/org/apache/cassandra/cql/QueryProcessor.java
b/src/java/org/apache/cassandra/cql/QueryProcessor.java
index f896f24..7eaa111 100644
--- a/src/java/org/apache/cassandra/cql/QueryProcessor.java
+++ b/src/java/org/apache/cassandra/cql/QueryProcessor.java
@@ -74,7 +74,7 @@ public class QueryProcessor
public static final String DEFAULT_KEY_NAME =
bufferToString(CFMetaData.DEFAULT_KEY_NAME);
private static Listorg.apache.cassandra.db.Row getSlice(CFMetaData
metadata, SelectStatement select, ListByteBuffer variables)
-throws InvalidRequestException, ReadTimeoutException,
UnavailableException, IsBootstrappingException
+throws InvalidRequestException, ReadTimeoutException,
UnavailableException, IsBootstrappingException, WriteTimeoutException
{
ListReadCommand commands = new ArrayListReadCommand();
http://git-wip-us.apache.org/repos/asf/cassandra/blob/12277516/src/java/org/apache/cassandra/db/ConsistencyLevel.java
--
diff --git a/src/java/org/apache/cassandra/db/ConsistencyLevel.java
b/src/java/org/apache/cassandra/db/ConsistencyLevel.java
index 9f4fc49..aec3c2d 100644
---
[jira] [Commented] (CASSANDRA-5051) Allow automatic cleanup after gc_grace
[ https://issues.apache.org/jira/browse/CASSANDRA-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645832#comment-13645832 ] Jonathan Ellis commented on CASSANDRA-5051: --- bq. Why do half a dozen tests need initServer now? Via IRC: since we don't have have a token without it, and the bootstrap is not true, and those tests do compactions. Allow automatic cleanup after gc_grace -- Key: CASSANDRA-5051 URL: https://issues.apache.org/jira/browse/CASSANDRA-5051 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Brandon Williams Assignee: Vijay Labels: vnodes Fix For: 2.0 Attachments: 0001-5051-v4.patch, 0001-5051-v6.patch, 0001-5051-with-test-fixes.patch, 0001-CASSANDRA-5051.patch, 0002-5051-remove-upgradesstable.patch, 0002-5051-remove-upgradesstable-v4.patch, 0004-5051-additional-test-v4.patch, 5051-v2.txt When using vnodes, after adding a new node you have to run cleanup on all the machines, because you don't know which are affected and chances are it was most if not all of them. As an alternative to this intensive process, we could allow cleanup during compaction if the data is older than gc_grace (or perhaps some other time period since people tend to use gc_grace hacks to get rid of tombstones.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4316) Compaction Throttle too bursty with large rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645837#comment-13645837 ] Yuki Morishita commented on CASSANDRA-4316: --- +1 on 1.2 patch. You need license header for new classes though. Compaction Throttle too bursty with large rows -- Key: CASSANDRA-4316 URL: https://issues.apache.org/jira/browse/CASSANDRA-4316 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.0 Reporter: Wayne Lewis Assignee: Jonathan Ellis Fix For: 1.2.5 Attachments: 4316-1.2.txt, 4316-1.2.txt, 4316-1.2-v2.txt, 4316-v3.txt In org.apache.cassandra.db.compaction.CompactionIterable the check for compaction throttling occurs once every 1000 rows. In our workload this is much too large as we have many large rows (16 - 100 MB). With a 100 MB row, about 100 GB is read (and possibly written) before the compaction throttle sleeps. This causes bursts of essentially unthrottled compaction IO followed by a long sleep which yields inconsistence performance and high error rates during the bursts. We applied a workaround to check throttle every row which solved our performance and error issues: line 116 in org.apache.cassandra.db.compaction.CompactionIterable: if ((row++ % 1000) == 0) replaced with if ((row++ % 1) == 0) I think the better solution is to calculate how often throttle should be checked based on the throttle rate to apply sleeps more consistently. E.g. if 16MB/sec is the limit then check for sleep after every 16MB is read so sleeps are spaced out about every second. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5489) Fix 2.0 key and column aliases serialization and cqlsh DESC SCHEMA
[
https://issues.apache.org/jira/browse/CASSANDRA-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645845#comment-13645845
]
Aleksey Yeschenko commented on CASSANDRA-5489:
--
All right. I guess adding an all-or-nothing restriction at this point is
okay-ish and makes more sense than the current one-alias-at-a-time restriction
that's not even throwing a reasonable exception.
But I don't exactly see how this avoids having to handle nulls, unless you are
suggesting to serialize key/column aliases lists as '[]' in cases where there
have been no renames and not '[null, null, null]' as it is now in trunk (for
the 'test' table in your example). If that's the case, then I agree and support
you. If it's not, then I'm against it - having nulls in
column_aliases/key_aliases breaks DESCRIBE SCHEMA in cqlsh and every other
implementation.
Also, we still need to apply the cqlsh part of the v1 patch (and it can
actually to to 1.2, with an added benefit that 1.2 cqlsh will be able to
properly describe 2.0 schema).
P.S. ALTER TABLE RENAME is broken for metadata-less (created via thrift/CLI)
column families in trunk right now:
{noformat}
cqlsh:test ALTER TABLE test RENAME column2 TO foo;
Bad Request: Cannot rename unknown column column2 in table test
cqlsh:test ALTER TABLE test RENAME key TO foo;
Bad Request: Cannot rename unknown column key in table test
cqlsh:test ALTER TABLE test RENAME value TO foo;
Bad Request: Cannot rename unknown column value in table test
{noformat}
INSERT/SELECT still handle the auto-aliases as they should, but ALTER TABLE
RENAME is broken. I'm surprised we don't have a dtest for this.
Fix 2.0 key and column aliases serialization and cqlsh DESC SCHEMA
--
Key: CASSANDRA-5489
URL: https://issues.apache.org/jira/browse/CASSANDRA-5489
Project: Cassandra
Issue Type: Bug
Components: Core, Tools
Affects Versions: 2.0
Reporter: Aleksey Yeschenko
Assignee: Aleksey Yeschenko
Priority: Minor
Fix For: 2.0
Attachments: 5489-1.2.txt, 5489.txt
CASSANDRA-5125 made a slight change to how key_aliases and column_aliases are
serialized in schema. Prior to that we never kept nulls in the the json
pseudo-lists. This does break cqlsh and probably breaks 1.2 nodes receiving
such migrations as well. The patch reverts this behavior and also slightly
modifies cqlsh itself to ignore non-regular columns from
system.schema_columns table.
This patch breaks nothing, since 2.0 already handles 1.2 non-null padded
alias lists.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5489) Fix 2.0 key and column aliases serialization and cqlsh DESC SCHEMA
[ https://issues.apache.org/jira/browse/CASSANDRA-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-5489: - Reviewer: iamaleksey (was: slebresne) Assignee: Sylvain Lebresne (was: Aleksey Yeschenko) Fix 2.0 key and column aliases serialization and cqlsh DESC SCHEMA -- Key: CASSANDRA-5489 URL: https://issues.apache.org/jira/browse/CASSANDRA-5489 Project: Cassandra Issue Type: Bug Components: Core, Tools Affects Versions: 2.0 Reporter: Aleksey Yeschenko Assignee: Sylvain Lebresne Priority: Minor Fix For: 2.0 Attachments: 5489-1.2.txt, 5489.txt CASSANDRA-5125 made a slight change to how key_aliases and column_aliases are serialized in schema. Prior to that we never kept nulls in the the json pseudo-lists. This does break cqlsh and probably breaks 1.2 nodes receiving such migrations as well. The patch reverts this behavior and also slightly modifies cqlsh itself to ignore non-regular columns from system.schema_columns table. This patch breaks nothing, since 2.0 already handles 1.2 non-null padded alias lists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
git commit: Mention deprecated blob-string syntax in 2.0 NEWS.txt
Updated Branches: refs/heads/trunk 56237c916 - 526f98f1f Mention deprecated blob-string syntax in 2.0 NEWS.txt Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/526f98f1 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/526f98f1 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/526f98f1 Branch: refs/heads/trunk Commit: 526f98f1fa050a02317e4b4f28534741fad86b58 Parents: 56237c9 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue Apr 30 22:17:49 2013 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue Apr 30 22:17:49 2013 +0300 -- NEWS.txt |4 1 files changed, 4 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/526f98f1/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index 31f30ab..c3c2fba 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -29,6 +29,10 @@ Upgrading - ASSUME command has been removed from cqlsh. Use CQL3 blobAsType() and typeAsBlob() conversion functions instead. See https://cassandra.apache.org/doc/cql3/CQL.html#blobFun for details. +- Inputing blobs as string constants is now fully deprecated in + favor of blob constants. Make sure to update your applications to use + the new syntax while you are still on 1.2 (which supports both string + and blob constants for blob input) before upgrading to 2.0. Operations --
[1/3] git commit: Prevent repair when protocol version does not match patch by yukim; reviewed by jbellis for CASSANDRA-5523
Updated Branches:
refs/heads/cassandra-1.2 4f5c84742 - fbc4bbcec
refs/heads/trunk 526f98f1f - a9d97e381
Prevent repair when protocol version does not match patch by yukim; reviewed by
jbellis for CASSANDRA-5523
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fbc4bbce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fbc4bbce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fbc4bbce
Branch: refs/heads/cassandra-1.2
Commit: fbc4bbcecf9e2a9b6f305031bb3086dd11fbc509
Parents: 4f5c847
Author: Yuki Morishita yu...@apache.org
Authored: Tue Apr 30 14:27:34 2013 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Tue Apr 30 14:27:34 2013 -0500
--
CHANGES.txt|1 +
.../cassandra/service/AntiEntropyService.java |9 ++---
2 files changed, 7 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fbc4bbce/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3922cb3..93198f0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -14,6 +14,7 @@
* Fix repair -snapshot not working (CASSANDRA-5512)
* Set isRunning flag later in binary protocol server (CASSANDRA-5467)
* Fix use of CQL3 functions with descencind clustering order (CASSANDRA-5472)
+ * Prevent repair when protocol version does not match (CASSANDRA-5523)
Merged from 1.1
* Add retry mechanism to OTC for non-droppable_verbs (CASSANDRA-5393)
* Use allocator information to improve memtable memory usage estimate
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fbc4bbce/src/java/org/apache/cassandra/service/AntiEntropyService.java
--
diff --git a/src/java/org/apache/cassandra/service/AntiEntropyService.java
b/src/java/org/apache/cassandra/service/AntiEntropyService.java
index eafab67..f41089e 100644
--- a/src/java/org/apache/cassandra/service/AntiEntropyService.java
+++ b/src/java/org/apache/cassandra/service/AntiEntropyService.java
@@ -668,10 +668,13 @@ public class AntiEntropyService
throw new IOException(message);
}
-if (MessagingService.instance().getVersion(endpoint)
MessagingService.VERSION_11 isSequential)
+// All endpoints should be on the same protocol version
+if (!MessagingService.instance().knowsVersion(endpoint) ||
MessagingService.instance().getVersion(endpoint) !=
MessagingService.current_version)
{
-logger.info(String.format([repair #%s] Cannot repair
using snapshots as node %s is pre-1.1, getName(), endpoint));
-return;
+String message = Cannot repair among different protocol
versions;
+differencingDone.signalAll();
+logger.error(String.format([repair #%s] , getName()) +
message);
+throw new IOException(message);
}
}
[2/3] git commit: Prevent repair when protocol version does not match patch by yukim; reviewed by jbellis for CASSANDRA-5523
Prevent repair when protocol version does not match patch by yukim; reviewed by
jbellis for CASSANDRA-5523
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fbc4bbce
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fbc4bbce
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fbc4bbce
Branch: refs/heads/trunk
Commit: fbc4bbcecf9e2a9b6f305031bb3086dd11fbc509
Parents: 4f5c847
Author: Yuki Morishita yu...@apache.org
Authored: Tue Apr 30 14:27:34 2013 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Tue Apr 30 14:27:34 2013 -0500
--
CHANGES.txt|1 +
.../cassandra/service/AntiEntropyService.java |9 ++---
2 files changed, 7 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fbc4bbce/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 3922cb3..93198f0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -14,6 +14,7 @@
* Fix repair -snapshot not working (CASSANDRA-5512)
* Set isRunning flag later in binary protocol server (CASSANDRA-5467)
* Fix use of CQL3 functions with descencind clustering order (CASSANDRA-5472)
+ * Prevent repair when protocol version does not match (CASSANDRA-5523)
Merged from 1.1
* Add retry mechanism to OTC for non-droppable_verbs (CASSANDRA-5393)
* Use allocator information to improve memtable memory usage estimate
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fbc4bbce/src/java/org/apache/cassandra/service/AntiEntropyService.java
--
diff --git a/src/java/org/apache/cassandra/service/AntiEntropyService.java
b/src/java/org/apache/cassandra/service/AntiEntropyService.java
index eafab67..f41089e 100644
--- a/src/java/org/apache/cassandra/service/AntiEntropyService.java
+++ b/src/java/org/apache/cassandra/service/AntiEntropyService.java
@@ -668,10 +668,13 @@ public class AntiEntropyService
throw new IOException(message);
}
-if (MessagingService.instance().getVersion(endpoint)
MessagingService.VERSION_11 isSequential)
+// All endpoints should be on the same protocol version
+if (!MessagingService.instance().knowsVersion(endpoint) ||
MessagingService.instance().getVersion(endpoint) !=
MessagingService.current_version)
{
-logger.info(String.format([repair #%s] Cannot repair
using snapshots as node %s is pre-1.1, getName(), endpoint));
-return;
+String message = Cannot repair among different protocol
versions;
+differencingDone.signalAll();
+logger.error(String.format([repair #%s] , getName()) +
message);
+throw new IOException(message);
}
}
[3/3] git commit: Merge branch 'cassandra-1.2' into trunk
Merge branch 'cassandra-1.2' into trunk Conflicts: src/java/org/apache/cassandra/service/ActiveRepairService.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a9d97e38 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a9d97e38 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a9d97e38 Branch: refs/heads/trunk Commit: a9d97e381e1ee33e7e4c0ff42f29890a0bd289da Parents: 526f98f fbc4bbc Author: Yuki Morishita yu...@apache.org Authored: Tue Apr 30 14:29:43 2013 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Tue Apr 30 14:29:43 2013 -0500 -- CHANGES.txt|1 + .../cassandra/service/ActiveRepairService.java |9 + 2 files changed, 10 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a9d97e38/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a9d97e38/src/java/org/apache/cassandra/service/ActiveRepairService.java -- diff --cc src/java/org/apache/cassandra/service/ActiveRepairService.java index b692ab0,000..1752911 mode 100644,00..100644 --- a/src/java/org/apache/cassandra/service/ActiveRepairService.java +++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java @@@ -1,1135 -1,0 +1,1144 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * License); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.cassandra.service; + +import java.io.*; +import java.net.InetAddress; +import java.security.MessageDigest; +import java.util.*; +import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.locks.Condition; + +import com.google.common.base.Objects; +import com.google.common.collect.Sets; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.cassandra.concurrent.JMXConfigurableThreadPoolExecutor; +import org.apache.cassandra.concurrent.NamedThreadFactory; +import org.apache.cassandra.concurrent.Stage; +import org.apache.cassandra.concurrent.StageManager; +import org.apache.cassandra.config.DatabaseDescriptor; +import org.apache.cassandra.db.*; +import org.apache.cassandra.db.compaction.AbstractCompactedRow; +import org.apache.cassandra.db.compaction.CompactionManager; +import org.apache.cassandra.dht.AbstractBounds; +import org.apache.cassandra.dht.Range; +import org.apache.cassandra.dht.Token; +import org.apache.cassandra.gms.*; +import org.apache.cassandra.io.IVersionedSerializer; +import org.apache.cassandra.locator.TokenMetadata; +import org.apache.cassandra.net.*; +import org.apache.cassandra.streaming.StreamingRepairTask; +import org.apache.cassandra.utils.*; + +/** + * ActiveRepairService encapsulates validating (hashing) individual column families, + * exchanging MerkleTrees with remote nodes via a tree request/response conversation, + * and then triggering repairs for disagreeing ranges. + * + * The node where repair was invoked acts as the 'initiator,' where valid trees are sent after generation + * and where the local and remote tree will rendezvous in rendezvous(). + * Once the trees rendezvous, a Differencer is executed and the service can trigger repairs + * for disagreeing ranges. + * + * Tree comparison and repair triggering occur in the single threaded Stage.ANTI_ENTROPY. + * + * The steps taken to enact a repair are as follows: + * 1. A repair is requested via JMX/nodetool: + * * The initiator sends TreeRequest messages to all neighbors of the target node: when a node + * receives a TreeRequest, it will perform a validation (read-only) compaction to immediately validate + * the column family. This is performed on the CompactionManager ExecutorService. + * 2. The validation process builds the merkle tree by: + * * Calling Validator.prepare(), which samples the column family to determine key
[jira] [Resolved] (CASSANDRA-5520) Query tracing session info inconsistent with events info
[ https://issues.apache.org/jira/browse/CASSANDRA-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams resolved CASSANDRA-5520. - Resolution: Invalid bq. However, the event-level breakdown shows no such large duration It doesn't need to, the entire request time just needs to exceed the rpc timeout. Query tracing session info inconsistent with events info Key: CASSANDRA-5520 URL: https://issues.apache.org/jira/browse/CASSANDRA-5520 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Environment: Linux Reporter: Ilya Kirnos Session info for a trace is showing that a query took 10 seconds (it timed out). cqlsh:system_traces select session_id, duration, request from sessions where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | duration | request --+--+ c7e36a30-af3a-11e2-9ec9-772ec39805fe | 1230 | multiget_slice However, the event-level breakdown shows no such large duration: cqlsh:system_traces select * from events where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | event_id | activity | source | source_elapsed | thread --+--+--+++ c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a30-af3a-11e2-9480-e9d811e0fc18 | Message received from /50.112.90.147 |50.112.4.16 | 19 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a31-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.252.153.16 | 50.112.90.147 | 246 | WRITE-/50.112.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9480-e9d811e0fc18 | Message received from /50.112.90.147 |50.112.4.16 | 259 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.248.106.37 | 50.112.90.147 | 253 |WRITE-/50.112.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-b8dc-a7032a583115 | Message received from /50.112.90.147 | 50.112.213.136 | 25 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash |50.112.4.16 | 421 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.252.151.214 | 50.112.90.147 | 310 | WRITE-/50.112.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-b8dc-a7032a583115 | Message received from /50.112.90.147 | 50.112.213.136 | 106 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable references |50.112.4.16 | 444 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.248.106.37 | 50.112.90.147 | 352 |WRITE-/50.112.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | 50.112.213.136 | 144 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9480-e9d811e0fc18 |Merging memtable contents |50.112.4.16 | 472 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.248.95.237 | 50.112.90.147 | 362 | WRITE-/50.112.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | 50.112.213.136 | 164 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9480-e9d811e0fc18 | Merging data from memtables and 0 sstables |50.112.4.16 | 510 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.252.151.214 | 50.112.90.147 | 376 | WRITE-/50.112.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-b8dc-a7032a583115 |Merging memtable contents | 50.112.213.136 | 195 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9480-e9d811e0fc18
[1/6] git commit: Add mention of 4321/4411 to NEWS
Updated Branches: refs/heads/cassandra-1.1 50f49a640 - b6730aaa5 refs/heads/cassandra-1.2 fbc4bbcec - 927c4a4f9 refs/heads/trunk a9d97e381 - d5061a4bc Add mention of 4321/4411 to NEWS Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b6730aaa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b6730aaa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b6730aaa Branch: refs/heads/cassandra-1.1 Commit: b6730aaa5901b70b69a805ce36a5693498013523 Parents: 50f49a6 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Apr 30 15:19:53 2013 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Apr 30 15:19:58 2013 -0500 -- NEWS.txt |3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b6730aaa/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index f3511b7..a768ddd 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -109,8 +109,7 @@ Features Upgrading - -- Nothing specific to this release, but please see 1.1 if you are upgrading - from a previous version. +- If you have column families using the LeveledCompactionStrategy, you should run scrub on those column families. Features
[3/6] git commit: Add mention of 4321/4411 to NEWS
Add mention of 4321/4411 to NEWS Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b6730aaa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b6730aaa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b6730aaa Branch: refs/heads/trunk Commit: b6730aaa5901b70b69a805ce36a5693498013523 Parents: 50f49a6 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Apr 30 15:19:53 2013 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Apr 30 15:19:58 2013 -0500 -- NEWS.txt |3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b6730aaa/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index f3511b7..a768ddd 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -109,8 +109,7 @@ Features Upgrading - -- Nothing specific to this release, but please see 1.1 if you are upgrading - from a previous version. +- If you have column families using the LeveledCompactionStrategy, you should run scrub on those column families. Features
[2/6] git commit: Add mention of 4321/4411 to NEWS
Add mention of 4321/4411 to NEWS Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b6730aaa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b6730aaa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b6730aaa Branch: refs/heads/cassandra-1.2 Commit: b6730aaa5901b70b69a805ce36a5693498013523 Parents: 50f49a6 Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Apr 30 15:19:53 2013 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Apr 30 15:19:58 2013 -0500 -- NEWS.txt |3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b6730aaa/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index f3511b7..a768ddd 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -109,8 +109,7 @@ Features Upgrading - -- Nothing specific to this release, but please see 1.1 if you are upgrading - from a previous version. +- If you have column families using the LeveledCompactionStrategy, you should run scrub on those column families. Features
[5/6] git commit: Merge branch 'cassandra-1.1' into cassandra-1.2
Merge branch 'cassandra-1.1' into cassandra-1.2 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/927c4a4f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/927c4a4f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/927c4a4f Branch: refs/heads/cassandra-1.2 Commit: 927c4a4f98c9517eb84dbbdc8d500670332e79c1 Parents: fbc4bbc b6730aa Author: Brandon Williams brandonwilli...@apache.org Authored: Tue Apr 30 15:20:14 2013 -0500 Committer: Brandon Williams brandonwilli...@apache.org Committed: Tue Apr 30 15:20:14 2013 -0500 -- NEWS.txt |3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/927c4a4f/NEWS.txt --
[jira] [Commented] (CASSANDRA-5520) Query tracing session info inconsistent with events info
[ https://issues.apache.org/jira/browse/CASSANDRA-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645945#comment-13645945 ] Ilya Kirnos commented on CASSANDRA-5520: just to clarify: how can i tell which part of the request caused the timeout? or are you saying that all the parts, when added up, exceed 10 seconds? looking at the trace above, that's not the case. Query tracing session info inconsistent with events info Key: CASSANDRA-5520 URL: https://issues.apache.org/jira/browse/CASSANDRA-5520 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Environment: Linux Reporter: Ilya Kirnos Session info for a trace is showing that a query took 10 seconds (it timed out). cqlsh:system_traces select session_id, duration, request from sessions where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | duration | request --+--+ c7e36a30-af3a-11e2-9ec9-772ec39805fe | 1230 | multiget_slice However, the event-level breakdown shows no such large duration: cqlsh:system_traces select * from events where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | event_id | activity | source | source_elapsed | thread --+--+--+++ c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a30-af3a-11e2-9480-e9d811e0fc18 | Message received from /50.112.90.147 |50.112.4.16 | 19 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a31-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.252.153.16 | 50.112.90.147 | 246 | WRITE-/50.112.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9480-e9d811e0fc18 | Message received from /50.112.90.147 |50.112.4.16 | 259 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.248.106.37 | 50.112.90.147 | 253 |WRITE-/50.112.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-b8dc-a7032a583115 | Message received from /50.112.90.147 | 50.112.213.136 | 25 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash |50.112.4.16 | 421 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.252.151.214 | 50.112.90.147 | 310 | WRITE-/50.112.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-b8dc-a7032a583115 | Message received from /50.112.90.147 | 50.112.213.136 | 106 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable references |50.112.4.16 | 444 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.248.106.37 | 50.112.90.147 | 352 |WRITE-/50.112.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | 50.112.213.136 | 144 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9480-e9d811e0fc18 |Merging memtable contents |50.112.4.16 | 472 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9ec9-772ec39805fe |Sending message to /10.248.95.237 | 50.112.90.147 | 362 | WRITE-/50.112.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | 50.112.213.136 | 164 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9480-e9d811e0fc18 | Merging data from memtables and 0 sstables |50.112.4.16 | 510 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.252.151.214 | 50.112.90.147 | 376 | WRITE-/50.112.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-b8dc-a7032a583115 |Merging memtable contents | 50.112.213.136 | 195 | ReadStage:11
[1/6] git commit: ant rat-write
Updated Branches:
refs/heads/cassandra-1.2 927c4a4f9 - 60f09f012
refs/heads/trunk d5061a4bc - 74f37b5a3
ant rat-write
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/60f09f01
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/60f09f01
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/60f09f01
Branch: refs/heads/cassandra-1.2
Commit: 60f09f0121e0801851b9ab017eddf7e326fa05fb
Parents: 2b0797b
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Apr 30 14:10:07 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Apr 30 15:36:57 2013 -0500
--
.../apache/cassandra/cache/IMeasurableMemory.java | 21 +
.../io/compress/CompressedThrottledReader.java | 21 +
.../apache/cassandra/io/util/ThrottledReader.java | 21 +
.../org/apache/cassandra/utils/ObjectSizes.java| 23 ++-
.../org/apache/cassandra/cache/ObjectSizeTest.java | 21 +
.../LegacyLeveledManifestTestHelper.java | 21 +
6 files changed, 127 insertions(+), 1 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/60f09f01/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
--
diff --git a/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
b/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
index 6b5f00e..16ca7c2 100644
--- a/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
+++ b/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
@@ -1,4 +1,25 @@
package org.apache.cassandra.cache;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
public interface IMeasurableMemory
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/60f09f01/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
--
diff --git
a/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
b/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
index 1b7b7a4..2c97a96 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
@@ -1,4 +1,25 @@
package org.apache.cassandra.io.compress;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
import java.io.File;
import java.io.FileNotFoundException;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/60f09f01/src/java/org/apache/cassandra/io/util/ThrottledReader.java
--
diff --git a/src/java/org/apache/cassandra/io/util/ThrottledReader.java
b/src/java/org/apache/cassandra/io/util/ThrottledReader.java
index d67550a..667f8ec 100644
--- a/src/java/org/apache/cassandra/io/util/ThrottledReader.java
+++ b/src/java/org/apache/cassandra/io/util/ThrottledReader.java
@@ -1,4 +1,25 @@
package org.apache.cassandra.io.util;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding
[2/6] git commit: fix compaction throttling bursty-ness patch by yukim and jbellis for CASSANDRA-4316
fix compaction throttling bursty-ness
patch by yukim and jbellis for CASSANDRA-4316
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2b0797b2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2b0797b2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2b0797b2
Branch: refs/heads/trunk
Commit: 2b0797b24e2d4a433c0e17506a0d8bb812f8f2dd
Parents: 927c4a4
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Apr 30 14:09:25 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Apr 30 15:36:57 2013 -0500
--
CHANGES.txt|1 +
.../db/compaction/AbstractCompactionStrategy.java |4 +-
.../db/compaction/CompactionController.java| 23 ++---
.../db/compaction/CompactionIterable.java | 12 ++---
.../cassandra/db/compaction/CompactionManager.java | 25 -
.../db/compaction/LeveledCompactionStrategy.java |7 ++-
.../db/compaction/ParallelCompactionIterable.java | 13 ++
.../apache/cassandra/db/compaction/Scrubber.java |8 +--
.../io/compress/CompressedRandomAccessReader.java |2 +-
.../io/compress/CompressedThrottledReader.java | 38 +++
.../io/sstable/SSTableBoundedScanner.java |6 ++-
.../apache/cassandra/io/sstable/SSTableReader.java | 25 +++---
.../cassandra/io/sstable/SSTableScanner.java |7 ++-
.../apache/cassandra/io/util/ThrottledReader.java | 35 +
.../org/apache/cassandra/tools/SSTableExport.java |5 +-
.../apache/cassandra/io/sstable/SSTableUtils.java |4 +-
16 files changed, 151 insertions(+), 64 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2b0797b2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 93198f0..bfece4f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2.5
+ * fix compaction throttling bursty-ness (CASSANDRA-4316)
* reduce memory consumption of IndexSummary (CASSANDRA-5506)
* remove per-row column name bloom filters (CASSANDRA-5492)
* Include fatal errors in trace events (CASSANDRA-5447)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2b0797b2/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
index a588216..636cb0d 100644
---
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
@@ -19,6 +19,7 @@ package org.apache.cassandra.db.compaction;
import java.util.*;
+import com.google.common.util.concurrent.RateLimiter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -150,9 +151,10 @@ public abstract class AbstractCompactionStrategy
*/
public ListICompactionScanner getScanners(CollectionSSTableReader
sstables, RangeToken range)
{
+RateLimiter limiter = CompactionManager.instance.getRateLimiter();
ArrayListICompactionScanner scanners = new
ArrayListICompactionScanner();
for (SSTableReader sstable : sstables)
-scanners.add(sstable.getDirectScanner(range));
+scanners.add(sstable.getDirectScanner(range, limiter));
return scanners;
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2b0797b2/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index f3198ff..f91c7a5 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -22,6 +22,8 @@ import java.util.Collections;
import java.util.List;
import java.util.Set;
+import com.google.common.primitives.Ints;
+import com.google.common.util.concurrent.RateLimiter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -36,6 +38,8 @@ import org.apache.cassandra.service.CacheService;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.utils.AlwaysPresentFilter;
import org.apache.cassandra.utils.Throttle;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.Throttle;
/**
* Manage compaction options.
@@ -50,20 +54,6 @@ public class CompactionController
public final int gcBefore;
public final int
[4/6] git commit: ant rat-write
ant rat-write
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/60f09f01
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/60f09f01
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/60f09f01
Branch: refs/heads/trunk
Commit: 60f09f0121e0801851b9ab017eddf7e326fa05fb
Parents: 2b0797b
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Apr 30 14:10:07 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Apr 30 15:36:57 2013 -0500
--
.../apache/cassandra/cache/IMeasurableMemory.java | 21 +
.../io/compress/CompressedThrottledReader.java | 21 +
.../apache/cassandra/io/util/ThrottledReader.java | 21 +
.../org/apache/cassandra/utils/ObjectSizes.java| 23 ++-
.../org/apache/cassandra/cache/ObjectSizeTest.java | 21 +
.../LegacyLeveledManifestTestHelper.java | 21 +
6 files changed, 127 insertions(+), 1 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/60f09f01/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
--
diff --git a/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
b/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
index 6b5f00e..16ca7c2 100644
--- a/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
+++ b/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
@@ -1,4 +1,25 @@
package org.apache.cassandra.cache;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
public interface IMeasurableMemory
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/60f09f01/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
--
diff --git
a/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
b/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
index 1b7b7a4..2c97a96 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
@@ -1,4 +1,25 @@
package org.apache.cassandra.io.compress;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
import java.io.File;
import java.io.FileNotFoundException;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/60f09f01/src/java/org/apache/cassandra/io/util/ThrottledReader.java
--
diff --git a/src/java/org/apache/cassandra/io/util/ThrottledReader.java
b/src/java/org/apache/cassandra/io/util/ThrottledReader.java
index d67550a..667f8ec 100644
--- a/src/java/org/apache/cassandra/io/util/ThrottledReader.java
+++ b/src/java/org/apache/cassandra/io/util/ThrottledReader.java
@@ -1,4 +1,25 @@
package org.apache.cassandra.io.util;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you
[3/6] git commit: fix compaction throttling bursty-ness patch by yukim and jbellis for CASSANDRA-4316
fix compaction throttling bursty-ness
patch by yukim and jbellis for CASSANDRA-4316
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2b0797b2
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2b0797b2
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2b0797b2
Branch: refs/heads/cassandra-1.2
Commit: 2b0797b24e2d4a433c0e17506a0d8bb812f8f2dd
Parents: 927c4a4
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Apr 30 14:09:25 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Apr 30 15:36:57 2013 -0500
--
CHANGES.txt|1 +
.../db/compaction/AbstractCompactionStrategy.java |4 +-
.../db/compaction/CompactionController.java| 23 ++---
.../db/compaction/CompactionIterable.java | 12 ++---
.../cassandra/db/compaction/CompactionManager.java | 25 -
.../db/compaction/LeveledCompactionStrategy.java |7 ++-
.../db/compaction/ParallelCompactionIterable.java | 13 ++
.../apache/cassandra/db/compaction/Scrubber.java |8 +--
.../io/compress/CompressedRandomAccessReader.java |2 +-
.../io/compress/CompressedThrottledReader.java | 38 +++
.../io/sstable/SSTableBoundedScanner.java |6 ++-
.../apache/cassandra/io/sstable/SSTableReader.java | 25 +++---
.../cassandra/io/sstable/SSTableScanner.java |7 ++-
.../apache/cassandra/io/util/ThrottledReader.java | 35 +
.../org/apache/cassandra/tools/SSTableExport.java |5 +-
.../apache/cassandra/io/sstable/SSTableUtils.java |4 +-
16 files changed, 151 insertions(+), 64 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2b0797b2/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 93198f0..bfece4f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.2.5
+ * fix compaction throttling bursty-ness (CASSANDRA-4316)
* reduce memory consumption of IndexSummary (CASSANDRA-5506)
* remove per-row column name bloom filters (CASSANDRA-5492)
* Include fatal errors in trace events (CASSANDRA-5447)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2b0797b2/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
index a588216..636cb0d 100644
---
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
@@ -19,6 +19,7 @@ package org.apache.cassandra.db.compaction;
import java.util.*;
+import com.google.common.util.concurrent.RateLimiter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -150,9 +151,10 @@ public abstract class AbstractCompactionStrategy
*/
public ListICompactionScanner getScanners(CollectionSSTableReader
sstables, RangeToken range)
{
+RateLimiter limiter = CompactionManager.instance.getRateLimiter();
ArrayListICompactionScanner scanners = new
ArrayListICompactionScanner();
for (SSTableReader sstable : sstables)
-scanners.add(sstable.getDirectScanner(range));
+scanners.add(sstable.getDirectScanner(range, limiter));
return scanners;
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2b0797b2/src/java/org/apache/cassandra/db/compaction/CompactionController.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
index f3198ff..f91c7a5 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionController.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionController.java
@@ -22,6 +22,8 @@ import java.util.Collections;
import java.util.List;
import java.util.Set;
+import com.google.common.primitives.Ints;
+import com.google.common.util.concurrent.RateLimiter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -36,6 +38,8 @@ import org.apache.cassandra.service.CacheService;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.utils.AlwaysPresentFilter;
import org.apache.cassandra.utils.Throttle;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.Throttle;
/**
* Manage compaction options.
@@ -50,20 +54,6 @@ public class CompactionController
public final int gcBefore;
public final int
[6/6] git commit: ghetto merge from 1.2
ghetto merge from 1.2
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/74f37b5a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/74f37b5a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/74f37b5a
Branch: refs/heads/trunk
Commit: 74f37b5a398fea0d60c5dc87124208a28580e8c3
Parents: e74c13f
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Apr 30 15:51:34 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Apr 30 15:51:34 2013 -0500
--
CHANGES.txt|1 +
.../apache/cassandra/cache/IMeasurableMemory.java | 21 +++
.../db/compaction/AbstractCompactionStrategy.java |4 +-
.../db/compaction/CompactionController.java| 24 +---
.../db/compaction/CompactionIterable.java | 12 +---
.../cassandra/db/compaction/CompactionManager.java | 25 +++-
.../db/compaction/LeveledCompactionStrategy.java |7 +-
.../db/compaction/ParallelCompactionIterable.java | 13 +---
.../apache/cassandra/db/compaction/Scrubber.java |8 +--
.../io/compress/CompressedRandomAccessReader.java |2 +-
.../io/sstable/SSTableBoundedScanner.java |6 +-
.../apache/cassandra/io/sstable/SSTableReader.java | 24 +++-
.../cassandra/io/sstable/SSTableScanner.java |7 ++-
.../org/apache/cassandra/tools/SSTableExport.java | 44 +++---
.../org/apache/cassandra/utils/ObjectSizes.java| 23 +++-
.../org/apache/cassandra/cache/ObjectSizeTest.java | 21 +++
.../LegacyLeveledManifestTestHelper.java | 21 +++
.../cassandra/io/sstable/SSTableReaderTest.java|4 +-
18 files changed, 171 insertions(+), 96 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/74f37b5a/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 48cd54a..31054e3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -40,6 +40,7 @@
* Add binary protocol versioning (CASSANDRA-5436)
1.2.5
+ * fix compaction throttling bursty-ness (CASSANDRA-4316)
* reduce memory consumption of IndexSummary (CASSANDRA-5506)
* remove per-row column name bloom filters (CASSANDRA-5492)
* Include fatal errors in trace events (CASSANDRA-5447)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/74f37b5a/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
--
diff --git a/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
b/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
index 6b5f00e..16ca7c2 100644
--- a/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
+++ b/src/java/org/apache/cassandra/cache/IMeasurableMemory.java
@@ -1,4 +1,25 @@
package org.apache.cassandra.cache;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
public interface IMeasurableMemory
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/74f37b5a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
index aefacd6..ba89aa3 100644
---
a/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/AbstractCompactionStrategy.java
@@ -21,6 +21,7 @@ import java.util.*;
import com.google.common.base.Predicate;
import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.RateLimiter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -207,9 +208,10 @@ public abstract class AbstractCompactionStrategy
*/
public ListICompactionScanner getScanners(CollectionSSTableReader
sstables, RangeToken range)
{
+RateLimiter limiter =
git commit: add ThrottledReader classes
Updated Branches:
refs/heads/trunk 74f37b5a3 - faf75d4d5
add ThrottledReader classes
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/faf75d4d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/faf75d4d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/faf75d4d
Branch: refs/heads/trunk
Commit: faf75d4d5b99a11bd9708e75972104f887c1a46c
Parents: 74f37b5
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Apr 30 16:01:14 2013 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Apr 30 16:01:14 2013 -0500
--
.../io/compress/CompressedThrottledReader.java | 59 +++
.../apache/cassandra/io/util/ThrottledReader.java | 56 ++
2 files changed, 115 insertions(+), 0 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/faf75d4d/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
--
diff --git
a/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
b/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
new file mode 100644
index 000..1f05953
--- /dev/null
+++ b/src/java/org/apache/cassandra/io/compress/CompressedThrottledReader.java
@@ -0,0 +1,59 @@
+package org.apache.cassandra.io.compress;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+
+import java.io.File;
+import java.io.FileNotFoundException;
+
+import com.google.common.util.concurrent.RateLimiter;
+
+import org.apache.cassandra.io.util.PoolingSegmentedFile;
+import org.apache.cassandra.io.util.RandomAccessReader;
+
+public class CompressedThrottledReader extends CompressedRandomAccessReader
+{
+private final RateLimiter limiter;
+
+public CompressedThrottledReader(String file, CompressionMetadata
metadata, RateLimiter limiter) throws FileNotFoundException
+{
+super(file, metadata, null);
+this.limiter = limiter;
+}
+
+protected void reBuffer()
+{
+limiter.acquire(buffer.length);
+super.reBuffer();
+}
+
+public static CompressedThrottledReader open(String file,
CompressionMetadata metadata, RateLimiter limiter)
+{
+try
+{
+return new CompressedThrottledReader(file, metadata, limiter);
+}
+catch (FileNotFoundException e)
+{
+throw new RuntimeException(e);
+}
+}
+}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/faf75d4d/src/java/org/apache/cassandra/io/util/ThrottledReader.java
--
diff --git a/src/java/org/apache/cassandra/io/util/ThrottledReader.java
b/src/java/org/apache/cassandra/io/util/ThrottledReader.java
new file mode 100644
index 000..b12a8a8
--- /dev/null
+++ b/src/java/org/apache/cassandra/io/util/ThrottledReader.java
@@ -0,0 +1,56 @@
+package org.apache.cassandra.io.util;
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+
+import java.io.File;
+import java.io.FileNotFoundException;
+
+import com.google.common.util.concurrent.RateLimiter;
+
+public class ThrottledReader extends RandomAccessReader
+{
+private final RateLimiter limiter;
+
+
[jira] [Updated] (CASSANDRA-5520) Query tracing session info inconsistent with events info
[ https://issues.apache.org/jira/browse/CASSANDRA-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Kirnos updated CASSANDRA-5520: --- Description: Session info for a trace is showing that a query took 10 seconds (it timed out). cqlsh:system_traces select session_id, duration, request from sessions where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | duration | request c7e36a30-af3a-11e2-9ec9-772ec39805fe | 1230 | multiget_slice However, the event-level breakdown shows no such large duration: cqlsh:system_traces select * from events where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | event_id | activity | source | source_elapsed | thread -+--- c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a30-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 19 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a31-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.153.16 | xxx.xxx.90.147 | 246 | WRITE-/xxx.xxx.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 259 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 253 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 25 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash | xxx.xxx.4.16 | 421 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 310 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 106 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable references | xxx.xxx.4.16 | 444 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 352 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | xxx.xxx.213.136 | 144 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9480-e9d811e0fc18 | Merging memtable contents | xxx.xxx.4.16 | 472 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | xxx.xxx.90.147 | 362 | WRITE-/xxx.xxx.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | xxx.xxx.213.136 | 164 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9480-e9d811e0fc18 | Merging data from memtables and 0 sstables | xxx.xxx.4.16 | 510 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 376 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-b8dc-a7032a583115 | Merging memtable contents | xxx.xxx.213.136 | 195 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9480-e9d811e0fc18 | Read 0 live cells and 0 tombstoned | xxx.xxx.4.16 | 530 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | xxx.xxx.90.147 | 401 | WRITE-/xxx.xxx.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | xxx.xxx.213.136 | 202 | ReadStage:41 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39146-af3a-11e2-9480-e9d811e0fc18 | Enqueuing response to /xxx.xxx.90.147 | xxx.xxx.4.16 | 572 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39146-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.153.16 | xxx.xxx.90.147 | 489 | WRITE-/xxx.xxx.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39146-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | xxx.xxx.213.136 | 215 | ReadStage:41 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39147-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash | xxx.xxx.4.16 | 610 | ReadStage:5330 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39147-af3a-11e2-b8dc-a7032a583115 | Merging data from memtables and 0 sstables | xxx.xxx.213.136 | 239 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39148-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable
[jira] [Created] (CASSANDRA-5528) CLUSTERING ORDER BY support for cqlsh's DESCRIBE
Tyler Hobbs created CASSANDRA-5528:
--
Summary: CLUSTERING ORDER BY support for cqlsh's DESCRIBE
Key: CASSANDRA-5528
URL: https://issues.apache.org/jira/browse/CASSANDRA-5528
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Tyler Hobbs
Assignee: Tyler Hobbs
Priority: Minor
Attachments: 5528-clustering-order-v1.txt
cqlsh currently does not output any sort of {{CLUSTERING ORDER BY}} options
with {{DESCRIBE}} and, furthermore, {{DESC}} orderings will result in bad
column type definitions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5528) CLUSTERING ORDER BY support for cqlsh's DESCRIBE
[
https://issues.apache.org/jira/browse/CASSANDRA-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-5528:
---
Attachment: 5528-clustering-order-v1.txt
Attached patch adds support for {{CLUSTERING ORDER BY}} and fixes the column
type definition problem.
CLUSTERING ORDER BY support for cqlsh's DESCRIBE
Key: CASSANDRA-5528
URL: https://issues.apache.org/jira/browse/CASSANDRA-5528
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Tyler Hobbs
Assignee: Tyler Hobbs
Priority: Minor
Attachments: 5528-clustering-order-v1.txt
cqlsh currently does not output any sort of {{CLUSTERING ORDER BY}} options
with {{DESCRIBE}} and, furthermore, {{DESC}} orderings will result in bad
column type definitions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5528) CLUSTERING ORDER BY support for cqlsh's DESCRIBE
[
https://issues.apache.org/jira/browse/CASSANDRA-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-5528:
---
Attachment: cql3_test_cases
For what it's worth, {{cql3_test_cases}} includes the table definitions that I
used to test out the changes.
CLUSTERING ORDER BY support for cqlsh's DESCRIBE
Key: CASSANDRA-5528
URL: https://issues.apache.org/jira/browse/CASSANDRA-5528
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Tyler Hobbs
Assignee: Tyler Hobbs
Priority: Minor
Attachments: 5528-clustering-order-v1.txt, cql3_test_cases
cqlsh currently does not output any sort of {{CLUSTERING ORDER BY}} options
with {{DESCRIBE}} and, furthermore, {{DESC}} orderings will result in bad
column type definitions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5528) CLUSTERING ORDER BY support for cqlsh's DESCRIBE
[
https://issues.apache.org/jira/browse/CASSANDRA-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-5528:
-
Reviewer: iamaleksey
CLUSTERING ORDER BY support for cqlsh's DESCRIBE
Key: CASSANDRA-5528
URL: https://issues.apache.org/jira/browse/CASSANDRA-5528
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Tyler Hobbs
Assignee: Tyler Hobbs
Priority: Minor
Attachments: 5528-clustering-order-v1.txt, cql3_test_cases
cqlsh currently does not output any sort of {{CLUSTERING ORDER BY}} options
with {{DESCRIBE}} and, furthermore, {{DESC}} orderings will result in bad
column type definitions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5520) Query tracing session info inconsistent with events info
[ https://issues.apache.org/jira/browse/CASSANDRA-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646116#comment-13646116 ] Brandon Williams commented on CASSANDRA-5520: - The events may not always add up to the total. There are at least two ways sessions and events can be inconsistent with each other: first, there are events happening after the session has ended, and second, there are too few events to explain why the session ended so late. You see to have the latter case. Query tracing session info inconsistent with events info Key: CASSANDRA-5520 URL: https://issues.apache.org/jira/browse/CASSANDRA-5520 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Environment: Linux Reporter: Ilya Kirnos Session info for a trace is showing that a query took 10 seconds (it timed out). cqlsh:system_traces select session_id, duration, request from sessions where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | duration | request c7e36a30-af3a-11e2-9ec9-772ec39805fe | 1230 | multiget_slice However, the event-level breakdown shows no such large duration: cqlsh:system_traces select * from events where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | event_id | activity | source | source_elapsed | thread -+--- c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a30-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 19 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a31-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.153.16 | xxx.xxx.90.147 | 246 | WRITE-/xxx.xxx.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 259 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 253 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 25 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash | xxx.xxx.4.16 | 421 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 310 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 106 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable references | xxx.xxx.4.16 | 444 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 352 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | xxx.xxx.213.136 | 144 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9480-e9d811e0fc18 | Merging memtable contents | xxx.xxx.4.16 | 472 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | xxx.xxx.90.147 | 362 | WRITE-/xxx.xxx.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | xxx.xxx.213.136 | 164 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9480-e9d811e0fc18 | Merging data from memtables and 0 sstables | xxx.xxx.4.16 | 510 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 376 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-b8dc-a7032a583115 | Merging memtable contents | xxx.xxx.213.136 | 195 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9480-e9d811e0fc18 | Read 0 live cells and 0 tombstoned | xxx.xxx.4.16 | 530 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | xxx.xxx.90.147 | 401 | WRITE-/xxx.xxx.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | xxx.xxx.213.136 | 202 | ReadStage:41 c7e36a30-af3a-11e2-9ec9-772ec39805fe |
[jira] [Comment Edited] (CASSANDRA-5520) Query tracing session info inconsistent with events info
[ https://issues.apache.org/jira/browse/CASSANDRA-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646116#comment-13646116 ] Brandon Williams edited comment on CASSANDRA-5520 at 4/30/13 11:08 PM: --- The events may not always add up to the total. There are at least two ways sessions and events can be inconsistent with each other: first, there are events happening after the session has ended, and second, there are too few events to explain why the session ended so late. You seem to have the latter case. was (Author: brandon.williams): The events may not always add up to the total. There are at least two ways sessions and events can be inconsistent with each other: first, there are events happening after the session has ended, and second, there are too few events to explain why the session ended so late. You see to have the latter case. Query tracing session info inconsistent with events info Key: CASSANDRA-5520 URL: https://issues.apache.org/jira/browse/CASSANDRA-5520 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Environment: Linux Reporter: Ilya Kirnos Session info for a trace is showing that a query took 10 seconds (it timed out). cqlsh:system_traces select session_id, duration, request from sessions where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | duration | request c7e36a30-af3a-11e2-9ec9-772ec39805fe | 1230 | multiget_slice However, the event-level breakdown shows no such large duration: cqlsh:system_traces select * from events where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | event_id | activity | source | source_elapsed | thread -+--- c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a30-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 19 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a31-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.153.16 | xxx.xxx.90.147 | 246 | WRITE-/xxx.xxx.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 259 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 253 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 25 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash | xxx.xxx.4.16 | 421 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 310 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 106 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable references | xxx.xxx.4.16 | 444 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 352 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | xxx.xxx.213.136 | 144 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9480-e9d811e0fc18 | Merging memtable contents | xxx.xxx.4.16 | 472 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | xxx.xxx.90.147 | 362 | WRITE-/xxx.xxx.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | xxx.xxx.213.136 | 164 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9480-e9d811e0fc18 | Merging data from memtables and 0 sstables | xxx.xxx.4.16 | 510 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 376 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-b8dc-a7032a583115 | Merging memtable contents | xxx.xxx.213.136 | 195 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9480-e9d811e0fc18 | Read 0 live cells and 0 tombstoned |
[jira] [Commented] (CASSANDRA-5520) Query tracing session info inconsistent with events info
[ https://issues.apache.org/jira/browse/CASSANDRA-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646135#comment-13646135 ] Ilya Kirnos commented on CASSANDRA-5520: Sorry to be dense, but: what does it mean for a session to end late with no events seemingly in flight? isn't a session just a container for events? Query tracing session info inconsistent with events info Key: CASSANDRA-5520 URL: https://issues.apache.org/jira/browse/CASSANDRA-5520 Project: Cassandra Issue Type: Bug Affects Versions: 1.2.4 Environment: Linux Reporter: Ilya Kirnos Session info for a trace is showing that a query took 10 seconds (it timed out). cqlsh:system_traces select session_id, duration, request from sessions where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | duration | request c7e36a30-af3a-11e2-9ec9-772ec39805fe | 1230 | multiget_slice However, the event-level breakdown shows no such large duration: cqlsh:system_traces select * from events where session_id = c7e36a30-af3a-11e2-9ec9-772ec39805fe; session_id | event_id | activity | source | source_elapsed | thread -+--- c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a30-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 19 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e36a31-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.153.16 | xxx.xxx.90.147 | 246 | WRITE-/xxx.xxx.4.16 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9480-e9d811e0fc18 | Message received from /xxx.xxx.90.147 | xxx.xxx.4.16 | 259 | Thread-57 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 253 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39140-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 25 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9480-e9d811e0fc18 | Executing single-partition query on CardHash | xxx.xxx.4.16 | 421 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 310 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39141-af3a-11e2-b8dc-a7032a583115 | Message received from /xxx.xxx.90.147 | xxx.xxx.213.136 | 106 | Thread-94 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9480-e9d811e0fc18 | Acquiring sstable references | xxx.xxx.4.16 | 444 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.106.37 | xxx.xxx.90.147 | 352 | WRITE-/xxx.xxx.79.52 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39142-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | xxx.xxx.213.136 | 144 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9480-e9d811e0fc18 | Merging memtable contents | xxx.xxx.4.16 | 472 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | xxx.xxx.90.147 | 362 | WRITE-/xxx.xxx.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39143-af3a-11e2-b8dc-a7032a583115 | Acquiring sstable references | xxx.xxx.213.136 | 164 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9480-e9d811e0fc18 | Merging data from memtables and 0 sstables | xxx.xxx.4.16 | 510 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-9ec9-772ec39805fe | Sending message to /xxx.xxx.151.214 | xxx.xxx.90.147 | 376 | WRITE-/xxx.xxx.213.136 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39144-af3a-11e2-b8dc-a7032a583115 | Merging memtable contents | xxx.xxx.213.136 | 195 | ReadStage:11 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9480-e9d811e0fc18 | Read 0 live cells and 0 tombstoned | xxx.xxx.4.16 | 530 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-9ec9-772ec39805fe | Sending message to /10.248.95.237 | xxx.xxx.90.147 | 401 | WRITE-/xxx.xxx.201.218 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39145-af3a-11e2-b8dc-a7032a583115 | Executing single-partition query on CardHash | xxx.xxx.213.136 | 202 | ReadStage:41 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39146-af3a-11e2-9480-e9d811e0fc18 | Enqueuing response to /xxx.xxx.90.147 | xxx.xxx.4.16 | 572 | ReadStage:5329 c7e36a30-af3a-11e2-9ec9-772ec39805fe | c7e39146-af3a-11e2-9ec9-772ec39805fe |
[jira] [Commented] (CASSANDRA-5528) CLUSTERING ORDER BY support for cqlsh's DESCRIBE
[
https://issues.apache.org/jira/browse/CASSANDRA-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646165#comment-13646165
]
Aleksey Yeschenko commented on CASSANDRA-5528:
--
+1 to the first part of the patch (fixing ReversedType handling), but I have an
issue with the second part.
{noformat}if len(layout.primary_key_components)
len(layout.partition_key_components){noformat} is equivalent to simply
{noformat}if layout.column_aliases{noformat}
similarly,
{noformat}
num_partition_components = len(layout.partition_key_components)
clustering_components = layout.primary_key_components[num_partition_components:]
clustering_names = self.cql_protect_names(clustering_components)
{noformat}
is equivalent to just
{noformat}
clustering_names = self.cql_protect_names(layout.column_aliases)
{noformat}
also, I don't fully understand why
{noformat}
if layout.compact_storage and not issubclass(layout.comparator, CompositeType)
{noformat}
is not just
{noformat}
if not issubclass(layout.comparator, CompositeType)
{noformat}
I also think that we shouldn't show CLUSTERING ORDER BY at all when we have no
ReversedType in use in the comparator (everything's ASC), just as we don't show
durable_writes when describing a keyspace unless it's the non-default 'false'.
CLUSTERING ORDER BY support for cqlsh's DESCRIBE
Key: CASSANDRA-5528
URL: https://issues.apache.org/jira/browse/CASSANDRA-5528
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Tyler Hobbs
Assignee: Tyler Hobbs
Priority: Minor
Attachments: 5528-clustering-order-v1.txt, cql3_test_cases
cqlsh currently does not output any sort of {{CLUSTERING ORDER BY}} options
with {{DESCRIBE}} and, furthermore, {{DESC}} orderings will result in bad
column type definitions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5484) Support custom secondary indexes in CQL
[
https://issues.apache.org/jira/browse/CASSANDRA-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646244#comment-13646244
]
Aleksey Yeschenko commented on CASSANDRA-5484:
--
I was going for
{noformat}
CREATE INDEX ON table(column) WITH type = CUSTOM AND options =
{'class_name': 'MyCostomIndex', ...};
{noformat}
to allow parametrization for KEYS/COMPOSITES index in the future, but I can't
think of what we could actually want to parametrize with them, so scratch that.
Let's only leave parametrization for CUSTOM.
But for the sake of consistency with CREATE KEYSPACE and
compaction,compression,etc. options for CREATE TABLE, I'd like options to be a
map (and replace 'class_name' key with 'class' internally, also for
consistency), e.g.
{noformat}
CREATE INDEX ON table(column) WITH options = {'class':
'MyCustomIndex',...};
{noformat}
Not sure about CREATE CUSTOM INDEX vs. just treating CREATE INDEX with non-null
options as custom, implicitly. I believe that every time we add a keyword to
CQL, even an unreserved one, a kitten dies somewhere, so I'd like to avoid
doing that.
Support custom secondary indexes in CQL
---
Key: CASSANDRA-5484
URL: https://issues.apache.org/jira/browse/CASSANDRA-5484
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.4
Reporter: Benjamin Coverston
Assignee: Aleksey Yeschenko
Through thrift users can add custom secondary indexes to the column metadata.
The following syntax is used in PLSQL, and I think we could use something
similar.
CREATE INDEX NAME ON TABLE (COLUMN) [INDEXTYPE IS (TYPENAME)
[PARAMETERS (PARAM[, PARAM])]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5529) ColumnFamilyRecordReader fails for large datasets
Rob Timpe created CASSANDRA-5529:
Summary: ColumnFamilyRecordReader fails for large datasets
Key: CASSANDRA-5529
URL: https://issues.apache.org/jira/browse/CASSANDRA-5529
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Affects Versions: 1.2.4, 1.1.11
Reporter: Rob Timpe
When running mapreduce jobs that read directly from cassandra, the job will
sometimes fail with an exception like this:
java.lang.RuntimeException: com.rockmelt.org.apache.thrift.TException: Message
length exceeded: 40
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:329)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:109)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.rockmelt.org.apache.thrift.TException: Message length exceeded:
40
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.cassandra.thrift.Column.read(Column.java:528)
at
org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408)
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12422)
at
com.rockmelt.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:696)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:680)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:362)
... 16 more
In ColumnFamilyRecordReader#initialize, a TBinaryProtocol is created as follows:
TTransport transport =
ConfigHelper.getInputTransportFactory(conf).openTransport(socket, conf);
TBinaryProtocol binaryProtocol = new TBinaryProtocol(transport,
ConfigHelper.getThriftMaxMessageLength(conf));
client = new Cassandra.Client(binaryProtocol);
But each time a call to cassandra is made, checkReadLength(int length) is
called in TBinaryProtocol, which includes this:
readLength_ -= length;
if (readLength_ 0) {
throw new TException(Message length exceeded: + length);
}
The result is that readLength_ is decreased each time, until it goes negative
and exception is thrown. This will only happen if you're reading a lot of data
and your split size is large (which is maybe why people haven't noticed it
earlier). This happens regardless of whether you use wide row support.
I'm not sure what the right fix is. It seems like you could either reset the
length of TBinaryProtocol after each call or just use a new TBinaryProtocol
each time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5530) Switch from THSHAServer to TThreadedSelectorServer
T Jake Luciani created CASSANDRA-5530:
-
Summary: Switch from THSHAServer to TThreadedSelectorServer
Key: CASSANDRA-5530
URL: https://issues.apache.org/jira/browse/CASSANDRA-5530
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 2.0
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing a
set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5530) Switch from THSHAServer to TThreadedSelectorServer
[
https://issues.apache.org/jira/browse/CASSANDRA-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-5530:
--
Attachment: 5530.txt
Switch from THSHAServer to TThreadedSelectorServer
--
Key: CASSANDRA-5530
URL: https://issues.apache.org/jira/browse/CASSANDRA-5530
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 2.0
Attachments: 5530.txt
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5530) Switch from THSHAServer to TThreadedSelectorServer
[
https://issues.apache.org/jira/browse/CASSANDRA-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-5530:
--
Description:
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
was:
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing a
set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
Switch from THSHAServer to TThreadedSelectorServer
--
Key: CASSANDRA-5530
URL: https://issues.apache.org/jira/browse/CASSANDRA-5530
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 2.0
Attachments: 5530.txt
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
[jira] [Updated] (CASSANDRA-5530) Switch from THSHAServer to TThreadedSelectorServer
[
https://issues.apache.org/jira/browse/CASSANDRA-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-5530:
--
Description:
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedSelectorServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
was:
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
Switch from THSHAServer to TThreadedSelectorServer
--
Key: CASSANDRA-5530
URL: https://issues.apache.org/jira/browse/CASSANDRA-5530
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 2.0
Attachments: 5530.txt
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate :
git commit: handleStateLeft no longer needs the version - removed
Updated Branches:
refs/heads/trunk faf75d4d5 - 559674593
handleStateLeft no longer needs the version - removed
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/55967459
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/55967459
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/55967459
Branch: refs/heads/trunk
Commit: 559674593d836b1fd90d3e5f00bb47728a42f4cd
Parents: faf75d4
Author: Dave Brosius dbros...@apache.org
Authored: Tue Apr 30 21:57:33 2013 -0400
Committer: Dave Brosius dbros...@apache.org
Committed: Tue Apr 30 21:57:33 2013 -0400
--
.../apache/cassandra/service/StorageService.java |1 -
1 files changed, 0 insertions(+), 1 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/55967459/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index d720df2..c44dfb2 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -1497,7 +1497,6 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
{
assert pieces.length = 2;
CollectionToken tokens;
-Integer version = MessagingService.instance().getVersion(endpoint);
tokens = getTokensFor(endpoint, pieces[1]);
if (logger.isDebugEnabled())
[jira] [Commented] (CASSANDRA-5530) Switch from THSHAServer to TThreadedSelectorServer
[
https://issues.apache.org/jira/browse/CASSANDRA-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646314#comment-13646314
]
Jonathan Ellis commented on CASSANDRA-5530:
---
Should we rename the option or at least add a comment that it's not actually
using THSHA anymore? +1 otherwise.
Switch from THSHAServer to TThreadedSelectorServer
--
Key: CASSANDRA-5530
URL: https://issues.apache.org/jira/browse/CASSANDRA-5530
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 2.0
Attachments: 5530.txt
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedSelectorServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5530) Switch from THSHAServer to TThreadedSelectorServer
[
https://issues.apache.org/jira/browse/CASSANDRA-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646315#comment-13646315
]
T Jake Luciani commented on CASSANDRA-5530:
---
Well it's still HSHA, just the HA is not single threaded...
Switch from THSHAServer to TThreadedSelectorServer
--
Key: CASSANDRA-5530
URL: https://issues.apache.org/jira/browse/CASSANDRA-5530
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 2.0
Attachments: 5530.txt
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedSelectorServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5530) Switch from THSHAServer to TThreadedSelectorServer
[
https://issues.apache.org/jira/browse/CASSANDRA-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646349#comment-13646349
]
Jonathan Ellis commented on CASSANDRA-5530:
---
Fair enough. Ship it!
Switch from THSHAServer to TThreadedSelectorServer
--
Key: CASSANDRA-5530
URL: https://issues.apache.org/jira/browse/CASSANDRA-5530
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: T Jake Luciani
Assignee: T Jake Luciani
Fix For: 2.0
Attachments: 5530.txt
TThreadedSelectorServer is new in Thrift 0.9. It builds on HSHA by allowing
for a set of threads for IO and a set work request processing...
I've attached the performance numbers below. It's a lot closer to
TThreadedServer.
ThreadedServer (Default)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 14811
interval_key_rate : 14811
latency median: 1.7
latency 95th percentile : 5.3
latency 99.9th percentile : 142.6
Total operation time : 00:01:16
END
Read Averages from the middle 80% of values:
interval_op_rate : 16898
interval_key_rate : 16898
latency median: 2.2
latency 95th percentile : 8.5
latency 99.9th percentile : 165.7
Total operation time : 00:01:05
END
{code}
HSHA (CURRENT)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 8939
interval_key_rate : 8939
latency median: 5.0
latency 95th percentile : 10.1
latency 99.9th percentile : 105.4
Total operation time : 00:01:56
END
Read Averages from the middle 80% of values:
interval_op_rate : 9608
interval_key_rate : 9608
latency median: 5.1
latency 95th percentile : 7.7
latency 99.9th percentile : 51.6
Total operation time : 00:01:49
END
{code}
TThreadedSelectorServer (NEW)
{code}
Write Averages from the middle 80% of values:
interval_op_rate : 11640
interval_key_rate : 11640
latency median: 3.1
latency 95th percentile : 10.6
latency 99.9th percentile : 135.9
Total operation time : 00:01:30
END
Read Averages from the middle 80% of values:
interval_op_rate : 15247
interval_key_rate : 15247
latency median: 2.8
latency 95th percentile : 7.1
latency 99.9th percentile : 40.3
Total operation time : 00:01:06
END
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5521) move IndexSummary off heap
[ https://issues.apache.org/jira/browse/CASSANDRA-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646352#comment-13646352 ] Pavel Yaskevich commented on CASSANDRA-5521: Have we considered using vint encoding on those arrays as we keep them in memory anyway to minimize space consumption? move IndexSummary off heap -- Key: CASSANDRA-5521 URL: https://issues.apache.org/jira/browse/CASSANDRA-5521 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jonathan Ellis Assignee: Vijay Fix For: 2.0 IndexSummary can still use a lot of heap for narrow-row sstables. (It can also contribute to memory fragmentation because of the large arrays it creates.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5529) ColumnFamilyRecordReader fails for large datasets
[
https://issues.apache.org/jira/browse/CASSANDRA-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-5529:
--
Attachment: 5529.txt
Rob, your analysis looks spot on.
WTF. Creating a new TBinaryProtocol for each message would be pretty ludicrous.
The genesis of this readLength_ business is hidden in the murky archives of the
Thrift incubator svn repro. It looks to me like it's kind of a really ugly
hack for pre-Framed transports that could call setReadLength in between
messages based on some kind of per-application knowledge. Because I can't
think of any use for expiring a connection after X bytes otherwise.
I don't think we should be using it at all. Attached is a patch that rips it
out, on the Cassandra server side as well. I feel sorry for any poor bastard
who ever pulled his head out over Cassandra erroring out his connection
apparently randomly...
ColumnFamilyRecordReader fails for large datasets
-
Key: CASSANDRA-5529
URL: https://issues.apache.org/jira/browse/CASSANDRA-5529
Project: Cassandra
Issue Type: Bug
Components: Hadoop
Affects Versions: 1.1.11, 1.2.4
Reporter: Rob Timpe
Attachments: 5529.txt
When running mapreduce jobs that read directly from cassandra, the job will
sometimes fail with an exception like this:
java.lang.RuntimeException: com.rockmelt.org.apache.thrift.TException:
Message length exceeded: 40
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:329)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:109)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.rockmelt.org.apache.thrift.TException: Message length
exceeded: 40
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.cassandra.thrift.Column.read(Column.java:528)
at
org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408)
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12422)
at
com.rockmelt.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:696)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:680)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:362)
... 16 more
In ColumnFamilyRecordReader#initialize, a TBinaryProtocol is created as
follows:
TTransport transport =
ConfigHelper.getInputTransportFactory(conf).openTransport(socket, conf);
TBinaryProtocol binaryProtocol = new TBinaryProtocol(transport,
ConfigHelper.getThriftMaxMessageLength(conf));
client = new Cassandra.Client(binaryProtocol);
But each time a call to cassandra is made, checkReadLength(int length) is
called in TBinaryProtocol, which includes this:
readLength_ -= length;
if (readLength_ 0) {
throw new TException(Message length exceeded: + length);
}
The result is that readLength_ is decreased each time, until it goes negative
and exception is thrown. This will only happen if you're reading a lot of
data and your split size is
[jira] [Updated] (CASSANDRA-5529) ColumnFamilyRecordReader fails for large datasets
[
https://issues.apache.org/jira/browse/CASSANDRA-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-5529:
--
Component/s: API
ColumnFamilyRecordReader fails for large datasets
-
Key: CASSANDRA-5529
URL: https://issues.apache.org/jira/browse/CASSANDRA-5529
Project: Cassandra
Issue Type: Bug
Components: API, Hadoop
Affects Versions: 0.6
Reporter: Rob Timpe
Assignee: Jonathan Ellis
Fix For: 1.1.12, 1.2.5
Attachments: 5529.txt
When running mapreduce jobs that read directly from cassandra, the job will
sometimes fail with an exception like this:
java.lang.RuntimeException: com.rockmelt.org.apache.thrift.TException:
Message length exceeded: 40
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:329)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:109)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.rockmelt.org.apache.thrift.TException: Message length
exceeded: 40
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.cassandra.thrift.Column.read(Column.java:528)
at
org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408)
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12422)
at
com.rockmelt.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:696)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:680)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:362)
... 16 more
In ColumnFamilyRecordReader#initialize, a TBinaryProtocol is created as
follows:
TTransport transport =
ConfigHelper.getInputTransportFactory(conf).openTransport(socket, conf);
TBinaryProtocol binaryProtocol = new TBinaryProtocol(transport,
ConfigHelper.getThriftMaxMessageLength(conf));
client = new Cassandra.Client(binaryProtocol);
But each time a call to cassandra is made, checkReadLength(int length) is
called in TBinaryProtocol, which includes this:
readLength_ -= length;
if (readLength_ 0) {
throw new TException(Message length exceeded: + length);
}
The result is that readLength_ is decreased each time, until it goes negative
and exception is thrown. This will only happen if you're reading a lot of
data and your split size is large (which is maybe why people haven't noticed
it earlier). This happens regardless of whether you use wide row support.
I'm not sure what the right fix is. It seems like you could either reset the
length of TBinaryProtocol after each call or just use a new TBinaryProtocol
each time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-5529) ColumnFamilyRecordReader fails for large datasets
[ https://issues.apache.org/jira/browse/CASSANDRA-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646374#comment-13646374 ] Jonathan Ellis edited comment on CASSANDRA-5529 at 5/1/13 4:41 AM: --- Rob, your analysis looks spot on. WTF. Creating a new TBinaryProtocol for each message would be pretty ludicrous. The genesis of this readLength_ business is hidden in the murky archives of the Thrift incubator svn repro. It looks to me like it's kind of a really ugly hack for pre-Framed transports that could call setReadLength in between messages based on some kind of per-application knowledge. Because I can't think of any use for expiring a connection after X bytes otherwise. I don't think we should be using it at all. Attached is a patch that rips it out, on the Cassandra server side as well. I feel sorry for any poor bastard who ever pulled his hair out over Cassandra erroring out his connection apparently randomly... was (Author: jbellis): Rob, your analysis looks spot on. WTF. Creating a new TBinaryProtocol for each message would be pretty ludicrous. The genesis of this readLength_ business is hidden in the murky archives of the Thrift incubator svn repro. It looks to me like it's kind of a really ugly hack for pre-Framed transports that could call setReadLength in between messages based on some kind of per-application knowledge. Because I can't think of any use for expiring a connection after X bytes otherwise. I don't think we should be using it at all. Attached is a patch that rips it out, on the Cassandra server side as well. I feel sorry for any poor bastard who ever pulled his head out over Cassandra erroring out his connection apparently randomly... ColumnFamilyRecordReader fails for large datasets - Key: CASSANDRA-5529 URL: https://issues.apache.org/jira/browse/CASSANDRA-5529 Project: Cassandra Issue Type: Bug Components: API, Hadoop Affects Versions: 0.6 Reporter: Rob Timpe Assignee: Jonathan Ellis Fix For: 1.1.12, 1.2.5 Attachments: 5529.txt When running mapreduce jobs that read directly from cassandra, the job will sometimes fail with an exception like this: java.lang.RuntimeException: com.rockmelt.org.apache.thrift.TException: Message length exceeded: 40 at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:329) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:109) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547) at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: com.rockmelt.org.apache.thrift.TException: Message length exceeded: 40 at com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393) at com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363) at org.apache.cassandra.thrift.Column.read(Column.java:528) at org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507) at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408) at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12422) at com.rockmelt.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:696) at
[jira] [Updated] (CASSANDRA-5529) ColumnFamilyRecordReader fails for large datasets
[
https://issues.apache.org/jira/browse/CASSANDRA-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-5529:
--
Attachment: 5529-1.1.txt
Thought I was on the 1.1 branch when I wrote that patch but it was really 1.2.
Here it is against 1.1 as well.
ColumnFamilyRecordReader fails for large datasets
-
Key: CASSANDRA-5529
URL: https://issues.apache.org/jira/browse/CASSANDRA-5529
Project: Cassandra
Issue Type: Bug
Components: API, Hadoop
Affects Versions: 0.6
Reporter: Rob Timpe
Assignee: Jonathan Ellis
Fix For: 1.1.12, 1.2.5
Attachments: 5529-1.1.txt, 5529.txt
When running mapreduce jobs that read directly from cassandra, the job will
sometimes fail with an exception like this:
java.lang.RuntimeException: com.rockmelt.org.apache.thrift.TException:
Message length exceeded: 40
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:329)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:109)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.rockmelt.org.apache.thrift.TException: Message length
exceeded: 40
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.cassandra.thrift.Column.read(Column.java:528)
at
org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408)
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12422)
at
com.rockmelt.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:696)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:680)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:362)
... 16 more
In ColumnFamilyRecordReader#initialize, a TBinaryProtocol is created as
follows:
TTransport transport =
ConfigHelper.getInputTransportFactory(conf).openTransport(socket, conf);
TBinaryProtocol binaryProtocol = new TBinaryProtocol(transport,
ConfigHelper.getThriftMaxMessageLength(conf));
client = new Cassandra.Client(binaryProtocol);
But each time a call to cassandra is made, checkReadLength(int length) is
called in TBinaryProtocol, which includes this:
readLength_ -= length;
if (readLength_ 0) {
throw new TException(Message length exceeded: + length);
}
The result is that readLength_ is decreased each time, until it goes negative
and exception is thrown. This will only happen if you're reading a lot of
data and your split size is large (which is maybe why people haven't noticed
it earlier). This happens regardless of whether you use wide row support.
I'm not sure what the right fix is. It seems like you could either reset the
length of TBinaryProtocol after each call or just use a new TBinaryProtocol
each time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5529) ColumnFamilyRecordReader fails for large datasets
[
https://issues.apache.org/jira/browse/CASSANDRA-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646379#comment-13646379
]
Jonathan Ellis commented on CASSANDRA-5529:
---
Note to self: remove our hacked TBinaryProtocol entirely in trunk.
ColumnFamilyRecordReader fails for large datasets
-
Key: CASSANDRA-5529
URL: https://issues.apache.org/jira/browse/CASSANDRA-5529
Project: Cassandra
Issue Type: Bug
Components: API, Hadoop
Affects Versions: 0.6
Reporter: Rob Timpe
Assignee: Jonathan Ellis
Fix For: 1.1.12, 1.2.5
Attachments: 5529-1.1.txt, 5529.txt
When running mapreduce jobs that read directly from cassandra, the job will
sometimes fail with an exception like this:
java.lang.RuntimeException: com.rockmelt.org.apache.thrift.TException:
Message length exceeded: 40
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:329)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:109)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: com.rockmelt.org.apache.thrift.TException: Message length
exceeded: 40
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
com.rockmelt.org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.cassandra.thrift.Column.read(Column.java:528)
at
org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
at org.apache.cassandra.thrift.KeySlice.read(KeySlice.java:408)
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12422)
at
com.rockmelt.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:696)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:680)
at
org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:362)
... 16 more
In ColumnFamilyRecordReader#initialize, a TBinaryProtocol is created as
follows:
TTransport transport =
ConfigHelper.getInputTransportFactory(conf).openTransport(socket, conf);
TBinaryProtocol binaryProtocol = new TBinaryProtocol(transport,
ConfigHelper.getThriftMaxMessageLength(conf));
client = new Cassandra.Client(binaryProtocol);
But each time a call to cassandra is made, checkReadLength(int length) is
called in TBinaryProtocol, which includes this:
readLength_ -= length;
if (readLength_ 0) {
throw new TException(Message length exceeded: + length);
}
The result is that readLength_ is decreased each time, until it goes negative
and exception is thrown. This will only happen if you're reading a lot of
data and your split size is large (which is maybe why people haven't noticed
it earlier). This happens regardless of whether you use wide row support.
I'm not sure what the right fix is. It seems like you could either reset the
length of TBinaryProtocol after each call or just use a new TBinaryProtocol
each time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-5521) move IndexSummary off heap
[ https://issues.apache.org/jira/browse/CASSANDRA-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646352#comment-13646352 ] Pavel Yaskevich edited comment on CASSANDRA-5521 at 5/1/13 5:55 AM: Have we considered using vint encoding on those arrays as we keep them in memory anyway to minimize space consumption? Edit: i remember now why that is not a good idea :) I wonder though how what could memory footprint be if we use TreeMap inside and keys and offsets (in vint encoding) saved in native memory... was (Author: xedin): Have we considered using vint encoding on those arrays as we keep them in memory anyway to minimize space consumption? move IndexSummary off heap -- Key: CASSANDRA-5521 URL: https://issues.apache.org/jira/browse/CASSANDRA-5521 Project: Cassandra Issue Type: Bug Components: Core Reporter: Jonathan Ellis Assignee: Vijay Fix For: 2.0 IndexSummary can still use a lot of heap for narrow-row sstables. (It can also contribute to memory fragmentation because of the large arrays it creates.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
