[jira] [Updated] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-10643: Status: Ready to Commit (was: Patch Available) > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > Labels: lcs > Attachments: 10643-trunk-REV01.txt, 10643-trunk-REV02.txt, > 10643-trunk-REV03.txt > > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408038#comment-15408038 ] Vishy Kasar commented on CASSANDRA-10643: - Thanks Marcus. Bit surprised that it was not doing any compaction. The test that I added here LeveledCompactionStrategyTest#testTokenRangeCompaction ran for me before your changes. That said, your changes look good. Please commit. > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > Labels: lcs > Attachments: 10643-trunk-REV01.txt, 10643-trunk-REV02.txt, > 10643-trunk-REV03.txt > > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-10643: Status: Patch Available (was: Awaiting Feedback) > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > Labels: lcs > Attachments: 10643-trunk-REV01.txt, 10643-trunk-REV02.txt, > 10643-trunk-REV03.txt > > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406930#comment-15406930 ] Vishy Kasar commented on CASSANDRA-10643: - Thanks for the quick review Marcus. I have taken care of most of your feedback. * Ongoing compactions are cancelled * In CompactionManager, submitOnSSTables and submitTask were merged in to a single method. I have kept the submitOnSSTables public to keep parity with submitMaximal. * In ColumnFamilyStore#sstablesInBounds: ** Used the Set ** Used the View.sstablesInBounds ** I have kept the Range to keep parity with Repair > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > Labels: lcs > Attachments: 10643-trunk-REV01.txt, 10643-trunk-REV02.txt, > 10643-trunk-REV03.txt > > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-10643: Attachment: 10643-trunk-REV03.txt > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > Labels: lcs > Attachments: 10643-trunk-REV01.txt, 10643-trunk-REV02.txt, > 10643-trunk-REV03.txt > > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-10643: Attachment: 10643-trunk-REV02.txt Second version of the patch with a test and rebased to trunk as of today. > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > Labels: lcs > Attachments: 10643-trunk-REV01.txt, 10643-trunk-REV02.txt > > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-10643: Attachment: 10643-trunk-REV01.txt > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > Attachments: 10643-trunk-REV01.txt > > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10643) Implement compaction for a specific token range
Vishy Kasar created CASSANDRA-10643: --- Summary: Implement compaction for a specific token range Key: CASSANDRA-10643 URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 Project: Cassandra Issue Type: Improvement Components: Compaction Reporter: Vishy Kasar We see repeated cases in production (using LCS) where small number of users generate a large number repeated updates or tombstones. Reading data of such users brings in large amounts of data in to java process. Apart from the read itself being slow for the user, the excessive GC affects other users as well. Our solution so far is to move from LCS to SCS and back. This takes long and is an over kill if the number of outliers is small. For such cases, we can implement the point compaction of a token range. We make the nodetool compact take a starting and ending token range and compact all the SSTables that fall with in that range. We can refuse to compact if the number of sstables is beyond a max_limit. Example: nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar reassigned CASSANDRA-7019: -- Assignee: Vishy Kasar (was: Branimir Lambov) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Vishy Kasar > Labels: compaction > Fix For: 3.x > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7019) Improve tombstone compactions
[ https://issues.apache.org/jira/browse/CASSANDRA-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-7019: --- Assignee: Branimir Lambov (was: Vishy Kasar) > Improve tombstone compactions > - > > Key: CASSANDRA-7019 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7019 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Branimir Lambov > Labels: compaction > Fix For: 3.x > > > When there are no other compactions to do, we trigger a single-sstable > compaction if there is more than X% droppable tombstones in the sstable. > In this ticket we should try to include overlapping sstables in those > compactions to be able to actually drop the tombstones. Might only be doable > with LCS (with STCS we would probably end up including all sstables) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-10643) Implement compaction for a specific token range
[ https://issues.apache.org/jira/browse/CASSANDRA-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar reassigned CASSANDRA-10643: --- Assignee: Vishy Kasar > Implement compaction for a specific token range > --- > > Key: CASSANDRA-10643 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10643 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vishy Kasar >Assignee: Vishy Kasar > > We see repeated cases in production (using LCS) where small number of users > generate a large number repeated updates or tombstones. Reading data of such > users brings in large amounts of data in to java process. Apart from the read > itself being slow for the user, the excessive GC affects other users as well. > Our solution so far is to move from LCS to SCS and back. This takes long and > is an over kill if the number of outliers is small. For such cases, we can > implement the point compaction of a token range. We make the nodetool compact > take a starting and ending token range and compact all the SSTables that fall > with in that range. We can refuse to compact if the number of sstables is > beyond a max_limit. > Example: > nodetool -st 3948291562518219268 -et 3948291562518219269 compact keyspace > table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10168) CassandraAuthorizer.authorize must throw exception when lookup of any auth table fails
Vishy Kasar created CASSANDRA-10168: --- Summary: CassandraAuthorizer.authorize must throw exception when lookup of any auth table fails Key: CASSANDRA-10168 URL: https://issues.apache.org/jira/browse/CASSANDRA-10168 Project: Cassandra Issue Type: Bug Components: Core Reporter: Vishy Kasar Attachments: authorizer.patch The PermissionCache is refreshed in the background every permissions_update_interval_in_ms by looking up cassandra tables. This background refresher assumes the called methods to throw exception on cassandra failures. In such cases, it just serves the stale entry until the next refresh happens. CassandraAuthorizer.authorize is throwing exception when it fails to lookup system_auth.users table. However when lookup on system_auth.permissions table fails, it swallows the exception and returns PERMISSION.NONE. In that case, the cache thinks that permission was revoked for the user until the next refresh succeeds. All the requests to that user on that cassandra instance fail incorrectly till the next refresh succeeds. This is bad. CassandraAuthorizer.authorize must throw exception when lookup of any auth table fails. I have attached a patch for cassandra 2.0 branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10145) Change protocol to allow sending key space independent of query string
Vishy Kasar created CASSANDRA-10145: --- Summary: Change protocol to allow sending key space independent of query string Key: CASSANDRA-10145 URL: https://issues.apache.org/jira/browse/CASSANDRA-10145 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar Currently keyspace is either embedded in the query string or set through use keyspace on a connection by client driver. There are practical use cases where client user has query and keyspace independently. In order for that scenario to work, they will have to create one client session per keyspace or have to resort to some string replace hackery. It will be nice if protocol allowed sending keyspace separately from the query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10009) Surface KeyspaceNotPresent as a specific exception code
Vishy Kasar created CASSANDRA-10009: --- Summary: Surface KeyspaceNotPresent as a specific exception code Key: CASSANDRA-10009 URL: https://issues.apache.org/jira/browse/CASSANDRA-10009 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar Fix For: 2.0.x When key space is not present, the server sends back an exception code of INVALID. The underlying reason is buried in the exception message. We have specific use cases where we want to retry KeyspaceNotPresent exception. However only way to achieve this today is to parse the exception message which is not good. Please surface KeyspaceNotPresent as a specific exception code -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9590) Support for both encrypted and unencrypted native transport connections
[
https://issues.apache.org/jira/browse/CASSANDRA-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583700#comment-14583700
]
Vishy Kasar commented on CASSANDRA-9590:
Thanks, this is similar to one of the features we requested:
https://issues.apache.org/jira/browse/CASSANDRA-8803 : Implement transitional
mode in C* that will accept both encrypted and non-encrypted client traffic
Support for both encrypted and unencrypted native transport connections
---
Key: CASSANDRA-9590
URL: https://issues.apache.org/jira/browse/CASSANDRA-9590
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Stefan Podkowinski
Enabling encryption for native transport currently turns SSL exclusively on
or off for the opened socket. Migrating from plain to encrypted requires to
migrate all native clients as well and redeploy all of them at the same time
after starting the SSL enabled Cassandra nodes.
This patch would allow to start Cassandra with both an unencrypted and ssl
enabled native port. Clients can connect to either, based whether they
support ssl or not.
This has been implemented by introducing a new {{native_transport_port_ssl}}
config option.
There would be three scenarios:
* client encryption disabled: native_transport_port unencrypted, port_ssl not
used
* client encryption enabled, port_ssl not set: encrypted native_transport_port
* client encryption enabled and port_ssl set: native_transport_port
unencrypted, port_ssl encrypted
This approach would keep configuration behavior fully backwards compatible.
Patch proposal (tests will be added later in case people will speak out in
favor for the patch):
[Diff
trunk|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/optionalnativessl],
[Patch against
trunk|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/optionalnativessl.patch]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9230) Allow preparing multiple prepared statements at once
[ https://issues.apache.org/jira/browse/CASSANDRA-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-9230: --- Description: We have a few cases like this: 1. Large (40K) clients 2. Each client preparing the same 10 prepared statements at the start up and on reconnection to node 3. Small(ish) number (24) of cassandra nodes The statement need to be prepared on a casasndra node just once but currently it is prepared 40K times at startup. https://issues.apache.org/jira/browse/CASSANDRA-8831 will make the situation much better. A further optimization is to allow clients to create not-yet prepared statements in bulk.This way, client can prepare all the not yet prepared statements with one round trip to server. was: We have a few cases like this: 1. Large (40K) clients 2. Each client preparing the same 10 prepared statements at the start up and on reconnection to node 3. Small(ish) number (24) of cassandra nodes The statement need to be prepared on a casasndra node just once but currently it is prepared 40K times at startup. https://issues.apache.org/jira/browse/CASSANDRA-8831 will make the situation much better. A further optimization is to allow clients to create not yet prepared statements in bulk.This way, client can prepare all the not yet statements with one round trip to server. Allow preparing multiple prepared statements at once Key: CASSANDRA-9230 URL: https://issues.apache.org/jira/browse/CASSANDRA-9230 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Vishy Kasar Priority: Minor Labels: ponies We have a few cases like this: 1. Large (40K) clients 2. Each client preparing the same 10 prepared statements at the start up and on reconnection to node 3. Small(ish) number (24) of cassandra nodes The statement need to be prepared on a casasndra node just once but currently it is prepared 40K times at startup. https://issues.apache.org/jira/browse/CASSANDRA-8831 will make the situation much better. A further optimization is to allow clients to create not-yet prepared statements in bulk.This way, client can prepare all the not yet prepared statements with one round trip to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9230) Allow preparing multiple prepared statements at once
[ https://issues.apache.org/jira/browse/CASSANDRA-9230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509547#comment-14509547 ] Vishy Kasar commented on CASSANDRA-9230: Yes, in the case described above. We also have other use cases where a given client connects to a limited set of cassandra nodes. So we can not always assume that the first client has prepared all the statements on all the cassandra nodes. We also have cases where number of prepared statements is much larger. Bulk prepare helps in all such cases. Allow preparing multiple prepared statements at once Key: CASSANDRA-9230 URL: https://issues.apache.org/jira/browse/CASSANDRA-9230 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Vishy Kasar Priority: Minor Labels: ponies We have a few cases like this: 1. Large (40K) clients 2. Each client preparing the same 10 prepared statements at the start up and on reconnection to node 3. Small(ish) number (24) of cassandra nodes The statement need to be prepared on a casasndra node just once but currently it is prepared 40K times at startup. https://issues.apache.org/jira/browse/CASSANDRA-8831 will make the situation much better. A further optimization is to allow clients to create not yet prepared statements in bulk.This way, client can prepare all the not yet statements with one round trip to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9230) Allow preparing multiple prepared statements at once
Vishy Kasar created CASSANDRA-9230: -- Summary: Allow preparing multiple prepared statements at once Key: CASSANDRA-9230 URL: https://issues.apache.org/jira/browse/CASSANDRA-9230 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar We have a few cases like this: 1. Large (40K) clients 2. Each client preparing the same 10 prepared statements at the start up and on reconnection to node 3. Small(ish) number (24) of cassandra nodes The statement need to be prepared on a casasndra node just once but currently it is prepared 40K times at startup. https://issues.apache.org/jira/browse/CASSANDRA-8831 will make the situation much better. A further optimization is to allow clients to create not yet prepared statements in bulk.This way, client can prepare all the not yet statements with one round trip to server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9226) Log streamid at the trace level on sending request and receiving response
Vishy Kasar created CASSANDRA-9226: -- Summary: Log streamid at the trace level on sending request and receiving response Key: CASSANDRA-9226 URL: https://issues.apache.org/jira/browse/CASSANDRA-9226 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar Multiple requests are sent on the same connection and responses arrive in any order. It is hard to track single request/response pair even when the trace logs are turned on today because the trace logs only provide information up to connection as shown below. It will be super useful to also provide the streamid in the trace logs. 2015/04/17 14:02:43.334 TRACE [New I/O worker #447] com.datastax.driver.core.Connection - Connection[host1.domain.com/10.0.1.1:9042-1, inFlight=1, closed=false] request sent successfully 2015/04/17 14:02:43.337 TRACE [New I/O worker #447] com.datastax.driver.core.Connection - Connection[host1.domain.com/10.0.1.1:9042-1, inFlight=1, closed=false] received: ROWS [column1 (timeuuid)][value (blob)][writetime(value) (bigint)] streamid will be useful at-least in the 2 logs above. But if you know of other cases where it will be useful, please add it there as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9226) Log streamid at the trace level on sending request and receiving response
[ https://issues.apache.org/jira/browse/CASSANDRA-9226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507677#comment-14507677 ] Vishy Kasar commented on CASSANDRA-9226: Sorry wrong project. This was meant to be a driver enhancement request. Will raise it there. Log streamid at the trace level on sending request and receiving response - Key: CASSANDRA-9226 URL: https://issues.apache.org/jira/browse/CASSANDRA-9226 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar Multiple requests are sent on the same connection and responses arrive in any order. It is hard to track single request/response pair even when the trace logs are turned on today because the trace logs only provide information up to connection as shown below. It will be super useful to also provide the streamid in the trace logs. 2015/04/17 14:02:43.334 TRACE [New I/O worker #447] com.datastax.driver.core.Connection - Connection[host1.domain.com/10.0.1.1:9042-1, inFlight=1, closed=false] request sent successfully 2015/04/17 14:02:43.337 TRACE [New I/O worker #447] com.datastax.driver.core.Connection - Connection[host1.domain.com/10.0.1.1:9042-1, inFlight=1, closed=false] received: ROWS [column1 (timeuuid)][value (blob)][writetime(value) (bigint)] streamid will be useful at-least in the 2 logs above. But if you know of other cases where it will be useful, please add it there as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-9226) Log streamid at the trace level on sending request and receiving response
[ https://issues.apache.org/jira/browse/CASSANDRA-9226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar resolved CASSANDRA-9226. Resolution: Invalid Log streamid at the trace level on sending request and receiving response - Key: CASSANDRA-9226 URL: https://issues.apache.org/jira/browse/CASSANDRA-9226 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar Multiple requests are sent on the same connection and responses arrive in any order. It is hard to track single request/response pair even when the trace logs are turned on today because the trace logs only provide information up to connection as shown below. It will be super useful to also provide the streamid in the trace logs. 2015/04/17 14:02:43.334 TRACE [New I/O worker #447] com.datastax.driver.core.Connection - Connection[host1.domain.com/10.0.1.1:9042-1, inFlight=1, closed=false] request sent successfully 2015/04/17 14:02:43.337 TRACE [New I/O worker #447] com.datastax.driver.core.Connection - Connection[host1.domain.com/10.0.1.1:9042-1, inFlight=1, closed=false] received: ROWS [column1 (timeuuid)][value (blob)][writetime(value) (bigint)] streamid will be useful at-least in the 2 logs above. But if you know of other cases where it will be useful, please add it there as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9205) Allow per statement time outs or request cancel method
Vishy Kasar created CASSANDRA-9205: -- Summary: Allow per statement time outs or request cancel method Key: CASSANDRA-9205 URL: https://issues.apache.org/jira/browse/CASSANDRA-9205 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Cassandra lets user specify time outs for various operations globally in yaml. It would be great if we could set different timeouts for CQL statements in different contexts, rather than just having a global timeouts in yaml. We have client requests that need to time out in a short duration vs some maintenance requests that we know take long. The only choice we have now is to set the server time out to the highest needed. User can certainly do session.executeAsync on the client side and wait for certain time on the returned future. However when user cancels the future on time out, nothing is done on the server side. We have seen cases where cassandra replicas were going over thousands of tombstones and causing OOMs way after client timed out. This can be implemented either by passing the time out along with query to server or by providing a cancel method similar to http://docs.oracle.com/javase/6/docs/api/java/sql/Statement.html It is understood that server may not be able to timeout/cancel the requests in all cases. So this is a request to server to do it's best effort to timeout/cancel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8957) Move TRUNCATE from MODIFY to DROP permission group
Vishy Kasar created CASSANDRA-8957: -- Summary: Move TRUNCATE from MODIFY to DROP permission group Key: CASSANDRA-8957 URL: https://issues.apache.org/jira/browse/CASSANDRA-8957 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Cassandra currently has 6 permissions: ALTER: required for ALTER KEYSPCE, ALTER TABLE, CREATE INDEX, DROP INDEX AUTHORIZE: required for GRANT, REVOKE CREATE: required for CREATE KEYSPACE, CREATE TABLE DROP: required for DROP KEYSPACE, DROP TABLE MODIFY: required for INSERT, DELETE, UPDATE, TRUNCATE SELECT: required for SELECT It seems incorrect to lump TRUNCATE with INSERT, DELETE, UPDATE. Every normal user typically does INSERT, DELETE, UPDATE. However a normal user does not need TRUNCATE. We want to prevent normal user accidentally truncating their tables in production. It is better to group TRUNCATE with other destructive operations such as DROP KEYSPACE, DROP TABLE. Proposal: Move TRUNCATE from MODIFY to DROP permission group Proposed 6 permissions looks like this: ALTER: required for ALTER KEYSPCE, ALTER TABLE, CREATE INDEX, DROP INDEX AUTHORIZE: required for GRANT, REVOKE CREATE: required for CREATE KEYSPACE, CREATE TABLE DROP: required for DROP KEYSPACE, DROP TABLE, TRUNCATE MODIFY: required for INSERT, DELETE, UPDATE SELECT: required for SELECT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8803) Implement transitional mode in C* that will accept both encrypted and non-encrypted client traffic
Vishy Kasar created CASSANDRA-8803: -- Summary: Implement transitional mode in C* that will accept both encrypted and non-encrypted client traffic Key: CASSANDRA-8803 URL: https://issues.apache.org/jira/browse/CASSANDRA-8803 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar We have some non-secure clusters taking live traffic in production from active clients. We want to enable client to node encryption on these clusters. Once we set the client_encryption_options enabled to true in yaml and bounce a cassandra node in the ring, the existing clients that do not do SSL will fail to connect to that node. There does not seem to be a good way to roll this change with out taking an outage. Can we implement a transitional mode in C* that will accept both encrypted and non-encrypted client traffic? We would enable this during transition and turn it off after both server and client start talking SSL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8086) Cassandra should have ability to limit the number of native connections
[ https://issues.apache.org/jira/browse/CASSANDRA-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302022#comment-14302022 ] Vishy Kasar commented on CASSANDRA-8086: We need this patch applied to 2.0 branch build as well. Thanks! Cassandra should have ability to limit the number of native connections --- Key: CASSANDRA-8086 URL: https://issues.apache.org/jira/browse/CASSANDRA-8086 Project: Cassandra Issue Type: Bug Reporter: Vishy Kasar Assignee: Norman Maurer Fix For: 2.1.3 Attachments: 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c-2.0.patch, 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c.patch, 0001-CASSANDRA-8086-Allow-to-limit-the-number-of-native-c.txt We have a production cluster with 72 instances spread across 2 DCs. We have a large number ( ~ 40,000 ) of clients hitting this cluster. Client normally connects to 4 cassandra instances. Some event (we think it is a schema change on server side) triggered the client to establish connections to all cassandra instances of local DC. This brought the server to its knees. The client connections failed and client attempted re-connections. Cassandra should protect itself from such attack from client. Do we have any knobs to control the number of max connections? If not, we need to add that knob. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar reassigned CASSANDRA-8194: -- Assignee: Aleksey Yeschenko (was: Sam Tunnicliffe) Thanks for the new patch Sam. Assigning it to Aleksey Yeschenko. Reading from Auth table should not be in the request path - Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Assignee: Aleksey Yeschenko Priority: Minor Fix For: 2.0.12, 3.0 Attachments: 8194-V2.patch, 8194-V3.txt, 8194-V4.txt, 8194.patch, CacheTest2.java We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 28 more ERROR [Thrift:17232] 2014-10-24 05:06:51,004
[jira] [Commented] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14266988#comment-14266988 ] Vishy Kasar commented on CASSANDRA-8194: Sam, CassandraAuthorizer executes the SELECT permissions query at ConsistencyLevel.LOCAL_ONE. It is quite possible that the query lands up on a node that is busy doing java-GC and times out. In our production use case, we see these time outs routinely even though our replication factor for system_auth is 10. The next user request will trigger reload with the query hopefully landing on a less busy node and succeeding. Due to this, I prefer the behavior in 8194-V2.patch where it continues to serve the stale entry till the SELECT succeeds. Otherwise clients will be confused as to why they are getting an AUTH related failure when no AUTH changes have occurred in the cluster. Another option is to do a one time retry of SELECT against a different node and throw if that fails as well. That will mostly eliminate the case of some busy node causing auth failures. Let me know what you think of this. I agree with your point on dedicated executor here. I had not realized that StorageService.tasks operates out of a single thread. Reading from Auth table should not be in the request path - Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Assignee: Vishy Kasar Priority: Minor Fix For: 2.0.12, 3.0 Attachments: 8194-V2.patch, 8194-V3.txt, 8194.patch, CacheTest2.java We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at
[jira] [Assigned] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar reassigned CASSANDRA-8194: -- Assignee: Sam Tunnicliffe (was: Vishy Kasar) Reading from Auth table should not be in the request path - Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Assignee: Sam Tunnicliffe Priority: Minor Fix For: 2.0.12, 3.0 Attachments: 8194-V2.patch, 8194-V3.txt, 8194.patch, CacheTest2.java We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 28 more ERROR [Thrift:17232] 2014-10-24 05:06:51,004 CustomTThreadPoolServer.java (line 224) Error occurred during processing of message.
[jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-8194: --- Attachment: 8194-V2.patch Thanks for the quick review Aleksey. I have attached 8194-V2.patch that addresses the issues you raised. Let me know if you have any further comments before committing this patch. Reading from Auth table should not be in the request path - Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor Fix For: 2.0.12, 3.0 Attachments: 8194-V2.patch, 8194.patch We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 28 more ERROR [Thrift:17232] 2014-10-24 05:06:51,004
[jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-8194: --- Attachment: CacheTest2.java Jeremiah, An attempt will be made to update at each refreshAfterWrite interval as long as there are active requests. Even if some of those refresh attempts fail/time-out, there will be others attempted as long as there are active requests. Some of those attempts should succeed and refresh the cache. If there are no active requests, then we do not care if the entry in cache is stale or not. I have attached a simple CacheTest2.java that illustrates this. Let me know if this is still a concern. Reading from Auth table should not be in the request path - Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor Fix For: 2.0.12, 3.0 Attachments: 8194-V2.patch, 8194.patch, CacheTest2.java We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at
[jira] [Created] (CASSANDRA-8450) Cassandra node announces availability before it is ready to accept CQL clients
Vishy Kasar created CASSANDRA-8450: -- Summary: Cassandra node announces availability before it is ready to accept CQL clients Key: CASSANDRA-8450 URL: https://issues.apache.org/jira/browse/CASSANDRA-8450 Project: Cassandra Issue Type: Bug Components: Core Reporter: Vishy Kasar We see cases where on a restart, the cassandra node announces the availability to java driver clients before it is ready to accept CQL clients. The java-driver gets an onUp message, attempts to create a connection to the cassandra node which promptly rejects its connection. When there are large number of clients, this error is seen by all clients. We should delay the announcements until the node is ready to accept connections. server log: INFO [main] 2014-12-07 01:34:47,921 Server.java (line 156) Starting listening for CQL clients on host/ip:9042... client log: com.datastax.driver.core.TransportException: [host1/ip1:9042] Cannot connect at com.datastax.driver.core.Connection.init(Connection.java:105) at com.datastax.driver.core.PooledConnection.init(PooledConnection.java:35) at com.datastax.driver.core.Connection$Factory.open(Connection.java:522) at com.datastax.driver.core.HostConnectionPool.init(HostConnectionPool.java:86) at com.datastax.driver.core.SessionManager.replacePool(SessionManager.java:269) at com.datastax.driver.core.SessionManager.access$400(SessionManager.java:39) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:301) at com.datastax.driver.core.SessionManager$3.call(SessionManager.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused: host1/ip1:9042 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ... 3 more 2014-12-07 01:34:44,011 [Cassandra Java Driver worker-1070] - onUp address=ip1 host=host1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8236) Delay node up and node added notifications until native protocol server is started
[
https://issues.apache.org/jira/browse/CASSANDRA-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240463#comment-14240463
]
Vishy Kasar commented on CASSANDRA-8236:
We have few clients hitting this. Is it possible to fix this in 2.0.12?
Delay node up and node added notifications until native protocol server
is started
--
Key: CASSANDRA-8236
URL: https://issues.apache.org/jira/browse/CASSANDRA-8236
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Tyler Hobbs
Assignee: Brandon Williams
Fix For: 3.0
Attachments: 8236.txt
As discussed in CASSANDRA-7510, there is still a gap between when a node up
or node added notification may be sent to native protocol clients (in
response to a gossip event) and when the native protocol server is ready to
serve requests.
Everything in between the call to {{StorageService.instance.initServer()}}
and creation of the native server in {{CassandraDaemon.setup()}} contributes
to this delay, but waiting for Gossip to settle introduces the biggest delay.
We may need to introduce a STARTING gossip state for the period inbetween,
which is why this is scheduled for 3.0. If there's a better option, though,
it may make sense to put this in 2.1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-8194: --- Attachment: 8194.patch I have attached a straight forward patch. This resolution will help many of our cassandra users who are experiencing latency increase/time-outs when security is enabled. I request it be included in the next 2.0.X release as opposed to 3.0. Reading from Auth table should not be in the request path - Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor Fix For: 3.0 Attachments: 8194.patch We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 28 more ERROR [Thrift:17232] 2014-10-24
[jira] [Created] (CASSANDRA-8194) Reading from Auth table should not be in the request path
Vishy Kasar created CASSANDRA-8194: -- Summary: Reading from Auth table should not be in the request path Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Bug Reporter: Vishy Kasar Attachments: auth-failure.txt We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the attached trace. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-8194: --- Attachment: (was: auth-failure.txt) Reading from Auth table should not be in the request path - Key: CASSANDRA-8194 URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 Project: Cassandra Issue Type: Bug Reporter: Vishy Kasar Assignee: Vishy Kasar We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the attached trace. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-8194: --- Description: We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. We still have few thousand requests failing each day with the trace below. The reason for this is read cache request realizing that cached entry has expired and doing a blocking request to refresh cache. We should have cache refreshed periodically only in the back ground. The user request should simply look at the cache and not try to refresh it. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) at org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) ... 19 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 28 more ERROR [Thrift:17232] 2014-10-24 05:06:51,004 CustomTThreadPoolServer.java (line 224) Error occurred during processing of message. com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at
[jira] [Commented] (CASSANDRA-8163) Complete restriction of a user to given keyspace
[ https://issues.apache.org/jira/browse/CASSANDRA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14185862#comment-14185862 ] Vishy Kasar commented on CASSANDRA-8163: By restricting the query (as Sylvain suggested), is this some thing doable in 2.0.X? It is preventing us from sharing cluster with multiple users. Complete restriction of a user to given keyspace Key: CASSANDRA-8163 URL: https://issues.apache.org/jira/browse/CASSANDRA-8163 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar Priority: Minor We have a cluster like this: project1_keyspace table101 table102 project2_keyspace table201 table202 We have set up following users and grants: project1_user has all access to project1_keyspace project2_user has all access to project2_keyspace However project1_user can still do a 'describe schema' and get the schema for project2_keyspace as well. We do not want project1_user to have any knowledge for project2 in any way (cqlsh/java-driver etc) . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8165) Do not include tombstones in schema version computation
Vishy Kasar created CASSANDRA-8165: -- Summary: Do not include tombstones in schema version computation Key: CASSANDRA-8165 URL: https://issues.apache.org/jira/browse/CASSANDRA-8165 Project: Cassandra Issue Type: Bug Reporter: Vishy Kasar During 1.2.19 migration, we found the schema version mismatch issue. On digging further, Sankalp found this was due to inclusion of the tombstones by 1.2.19 due to increased gc_grace_seconds. It seems wrong to include ephemeral data like tombstones in MD5 computation. Can this be avoided? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8163) Complete restriction of a user to given keyspace
Vishy Kasar created CASSANDRA-8163: -- Summary: Complete restriction of a user to given keyspace Key: CASSANDRA-8163 URL: https://issues.apache.org/jira/browse/CASSANDRA-8163 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar We have a cluster like this: project1_keyspace table101 table102 project2_keyspace table201 table202 We have set up following users and grants: project1_user has all access to project1_keyspace project2_user has all access to project2_keyspace However project1_user can still do a 'describe schema' and get the schema for project2_keyspace as well. We do not want project1_user to have any knowledge for project2 in any way (cqlsh/java-driver etc) . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8016) Auth tables should use higher consistency level
[ https://issues.apache.org/jira/browse/CASSANDRA-8016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170277#comment-14170277 ] Vishy Kasar commented on CASSANDRA-8016: We need to fallback to CL.LOCAL_QUORUM also in case of CL.LOCAL_ONE failure (as opposed to miss) Auth tables should use higher consistency level --- Key: CASSANDRA-8016 URL: https://issues.apache.org/jira/browse/CASSANDRA-8016 Project: Cassandra Issue Type: Bug Reporter: T Jake Luciani Assignee: Aleksey Yeschenko Fix For: 2.0.11 The Auth code in Cassandra uses CL.ONE or CL.LOCAL_ONE except in the case of the superuser. Since the Auth keyspace is created with RF=1 the default experience is fine. However if you change to RF 1 suddenly the select statements are open to misses. We should change reads/writes in Auth, PasswordAuthenticator, CassandraAuthorizer to always use LOCAL_QUORUM/QUORUM. For reads we could optimize the code to start with CL.ONE and on a miss increase to CL.QUORUM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8086) Cassandra should have ability to limit the number of native connections
Vishy Kasar created CASSANDRA-8086: -- Summary: Cassandra should have ability to limit the number of native connections Key: CASSANDRA-8086 URL: https://issues.apache.org/jira/browse/CASSANDRA-8086 Project: Cassandra Issue Type: Bug Reporter: Vishy Kasar Assignee: sankalp kohli We have a production cluster with 72 instances spread across 2 DCs. We have a large number ( ~ 40,000 ) of clients hitting this cluster. Client normally connects to 4 cassandra instances. Some event (we think it is a schema change on server side) triggered the client to establish connections to all cassandra instances of local DC. This brought the server to its knees. The client connections failed and client attempted re-connections. Cassandra should protect itself from such attack from client. Do we have any knobs to control the number of max connections? If not, we need to add that knob. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-7894) Authorization fails occasionally even when user has permission
Vishy Kasar created CASSANDRA-7894: -- Summary: Authorization fails occasionally even when user has permission Key: CASSANDRA-7894 URL: https://issues.apache.org/jira/browse/CASSANDRA-7894 Project: Cassandra Issue Type: Bug Reporter: Vishy Kasar Production cluster uses org.apache.cassandra.auth.PasswordAuthenticator and org.apache.cassandra.auth.CassandraAuthorizer. The user has SELECT and MODIFY permissions. cqlsh list ALL ON ALL KEYSPACES; x_cassandra_user | all keyspaces | SELECT x_cassandra_user | all keyspaces | MODIFY Millions of requests from user go through fine. A very small percentage fail with following message sent back to hector client. Caused by: me.prettyprint.hector.api.exceptions.HInvalidRequestException: InvalidRequestException(why:User x_cassandra_user has no SELECT permission on table ks.cf or any of its parents) Any ideas now why this fails? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7894) Authorization fails occasionally even when user has permission
[ https://issues.apache.org/jira/browse/CASSANDRA-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123897#comment-14123897 ] Vishy Kasar commented on CASSANDRA-7894: Thanks Wei. What would be a good exception to search in this case? Authorization fails occasionally even when user has permission -- Key: CASSANDRA-7894 URL: https://issues.apache.org/jira/browse/CASSANDRA-7894 Project: Cassandra Issue Type: Bug Reporter: Vishy Kasar Production cluster uses org.apache.cassandra.auth.PasswordAuthenticator and org.apache.cassandra.auth.CassandraAuthorizer. The user has SELECT and MODIFY permissions. cqlsh list ALL ON ALL KEYSPACES; x_cassandra_user | all keyspaces | SELECT x_cassandra_user | all keyspaces | MODIFY Millions of requests from user go through fine. A very small percentage fail with following message sent back to hector client. Caused by: me.prettyprint.hector.api.exceptions.HInvalidRequestException: InvalidRequestException(why:User x_cassandra_user has no SELECT permission on table ks.cf or any of its parents) Any ideas now why this fails? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7894) Authorization fails occasionally even when user has permission
[ https://issues.apache.org/jira/browse/CASSANDRA-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124236#comment-14124236 ] Vishy Kasar commented on CASSANDRA-7894: You are correct. This was due to read time out error on the authorization table. What is a good way to resolve this? These permissions do not change often. How can we cache them for longer so we do not have to look up often? com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:681) at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:749) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3690) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3678) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) at com.google.common.cache.LocalCache.get(LocalCache.java:3990) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) at org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:315) at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.getResult(Cassandra.java:3425) at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.getResult(Cassandra.java:3413) ... 6 more Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) ... 25 more Authorization fails occasionally even when user has permission -- Key: CASSANDRA-7894 URL: https://issues.apache.org/jira/browse/CASSANDRA-7894 Project:
[jira] [Created] (CASSANDRA-7632) NPE in AutoSavingCache$Writer.deleteOldCacheFiles
Vishy Kasar created CASSANDRA-7632: -- Summary: NPE in AutoSavingCache$Writer.deleteOldCacheFiles Key: CASSANDRA-7632 URL: https://issues.apache.org/jira/browse/CASSANDRA-7632 Project: Cassandra Issue Type: Bug Components: Core Reporter: Vishy Kasar Priority: Minor Observed this NPE in one of our production cluster (2.0.9). Does not seem to be causing harm but good to resolve. ERROR [CompactionExecutor:1188] 2014-07-27 21:57:08,225 CassandraDaemon.java (line 199) Exception in thread Thread[CompactionExecutor:1188,1,main] clusterName=clouddb_p03 java.lang.NullPointerException at org.apache.cassandra.cache.AutoSavingCache$Writer.deleteOldCacheFiles(AutoSavingCache.java:265) at org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavingCache.java:195) at org.apache.cassandra.db.compaction.CompactionManager$10.run(CompactionManager.java:862) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7384) Collect metrics on queries by consistency level
Vishy Kasar created CASSANDRA-7384: -- Summary: Collect metrics on queries by consistency level Key: CASSANDRA-7384 URL: https://issues.apache.org/jira/browse/CASSANDRA-7384 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Vishy Kasar Assignee: sankalp kohli Priority: Minor We had cases where cassandra client users thought that they were doing queries at one consistency level but turned out to be not correct. It will be good to collect metrics on number of queries done at various consistency level on the server. See the equivalent JIRA on java driver: https://datastax-oss.atlassian.net/browse/JAVA-354 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7104) Unable to use cqlsh when client certificate authentication is enabled
Vishy Kasar created CASSANDRA-7104: -- Summary: Unable to use cqlsh when client certificate authentication is enabled Key: CASSANDRA-7104 URL: https://issues.apache.org/jira/browse/CASSANDRA-7104 Project: Cassandra Issue Type: New Feature Reporter: Vishy Kasar As per, http://www.datastax.com/documentation/cassandra/1.2/cassandra/security/secureCqlshSSL_t.html You cannot use cqlsh when client certificate authentication is enabled (require_client_auth=true). This seems like a major limitation if we can not use cqlsh when client auth is enabled. Can this limitation be resolved? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-6660) Make node tool command take a password file
Vishy Kasar created CASSANDRA-6660: -- Summary: Make node tool command take a password file Key: CASSANDRA-6660 URL: https://issues.apache.org/jira/browse/CASSANDRA-6660 Project: Cassandra Issue Type: Improvement Reporter: Vishy Kasar We are sending the jmx password in the clear to the node tool command in production. This is a security risk. Any one doing a 'ps' can see the clear password. Can we change the node tool command to also take a password file argument. This file will list the JMX user and passwords. Example below: cat /cassandra/run/10003004.jmxpasswd monitorRole abc controlRole def Based on the user name provided, node tool can pick up the right password. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6585) Make node tool exit code non zero when it fails to create snapshot
[ https://issues.apache.org/jira/browse/CASSANDRA-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872292#comment-13872292 ] Vishy Kasar commented on CASSANDRA-6585: Thanks for the patch Brandon. I am going to apply this to our 1.1 code base. Any concerns? Make node tool exit code non zero when it fails to create snapshot -- Key: CASSANDRA-6585 URL: https://issues.apache.org/jira/browse/CASSANDRA-6585 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Vishy Kasar Assignee: Brandon Williams Fix For: 1.2.14 Attachments: 6585.txt When node tool snapshot is invoked on a bootstrapping node, it does not create the snapshot as expected. However node tool returns a zero exit code in that case. Can we make the node tool return a non zero exit code when create snapshot fails? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (CASSANDRA-6585) Make node tool exit code non zero when it fails to create snapshot
Vishy Kasar created CASSANDRA-6585: -- Summary: Make node tool exit code non zero when it fails to create snapshot Key: CASSANDRA-6585 URL: https://issues.apache.org/jira/browse/CASSANDRA-6585 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Vishy Kasar Fix For: 1.2.14 When node tool snapshot is invoked on a bootstrapping node, it does not create the snapshot as expected. However node tool returns a zero exit code in that case. Can we make the node tool return a non zero exit code when create snapshot fails? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-5988) Make hint TTL customizable
[ https://issues.apache.org/jira/browse/CASSANDRA-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-5988: --- Attachment: 5988.txt Make hint TTL customizable -- Key: CASSANDRA-5988 URL: https://issues.apache.org/jira/browse/CASSANDRA-5988 Project: Cassandra Issue Type: Bug Components: Core Reporter: Oleg Kibirev Labels: patch Attachments: 5988.txt Currently time to live for stored hints is hardcoded to be gc_grace_seconds. This causes problems for applications using backdated deletes as a form of optimistic locking. Hints for updates made to the same data on which delete was attempted can persist for days, making it impossible to determine if delete succeeded by doing read(ALL) after a reasonable delay. We need a way to explicitly configure hint TTL, either through schema parameter or through a yaml file. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5988) Make hint TTL customizable
[ https://issues.apache.org/jira/browse/CASSANDRA-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13801114#comment-13801114 ] Vishy Kasar commented on CASSANDRA-5988: Attached the diff as a file 5988.txt Make hint TTL customizable -- Key: CASSANDRA-5988 URL: https://issues.apache.org/jira/browse/CASSANDRA-5988 Project: Cassandra Issue Type: Bug Components: Core Reporter: Oleg Kibirev Labels: patch Attachments: 5988.txt Currently time to live for stored hints is hardcoded to be gc_grace_seconds. This causes problems for applications using backdated deletes as a form of optimistic locking. Hints for updates made to the same data on which delete was attempted can persist for days, making it impossible to determine if delete succeeded by doing read(ALL) after a reasonable delay. We need a way to explicitly configure hint TTL, either through schema parameter or through a yaml file. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (CASSANDRA-6184) Need a nodetool command to purge hints
Vishy Kasar created CASSANDRA-6184: -- Summary: Need a nodetool command to purge hints Key: CASSANDRA-6184 URL: https://issues.apache.org/jira/browse/CASSANDRA-6184 Project: Cassandra Issue Type: Bug Components: Tools Environment: Production Reporter: Vishy Kasar Fix For: 1.2.11 Our operation requires us to purge the hints. The only way to do this currently is to truncate the system/HintsColumnFamily across our entire cluster. That is error prone. We want a purgehints [keyspace] [cfnames] option on the nodetool command. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (CASSANDRA-5815) NPE from migration manager
Vishy Kasar created CASSANDRA-5815: -- Summary: NPE from migration manager Key: CASSANDRA-5815 URL: https://issues.apache.org/jira/browse/CASSANDRA-5815 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.12 Reporter: Vishy Kasar Priority: Minor In one of our production clusters we see this error often. Looking through the source, Gossiper.instance.getEndpointStateForEndpoint(endpoint) is returning null for some end point. De we need any config change on our end to resolve this? In any case, cassandra should be updated to protect against this NPE. ERROR [OptionalTasks:1] 2013-07-24 13:40:38,972 AbstractCassandraDaemon.java (line 132) Exception in thread Thread[OptionalTasks:1,5,main] java.lang.NullPointerException at org.apache.cassandra.service.MigrationManager$1.run(MigrationManager.java:134) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) It turned out that the reason for NPE was we bootstrapped a node with the same token as another node. Cassandra should not throw an NPE here but log a meaningful error message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-5637) sstableloader does not seem to load all the data present in the provided sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar resolved CASSANDRA-5637. Resolution: Not A Problem This turned out to be an issue with how we were reading back the data. The limit on how many columns to return on a read was set to 100 and it was not showing the newly added columns. Nothing more to do on this. sstableloader does not seem to load all the data present in the provided sstables - Key: CASSANDRA-5637 URL: https://issues.apache.org/jira/browse/CASSANDRA-5637 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.10 Reporter: Vishy Kasar Attachments: sstable-loader-log.txt I loaded about 1 million records in to my local cassandra. I then attempted to load these records using the sstableloader to prod cassandra using command sstableloader -v -u xyz -pw abc -d st11p02me-hpbb050130-i6 /tmp/prod_p02/containers When I read back the production data later, I do not find some rows that I inserted above. Few other rows are present though. Are there known bugs in sstableloader? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5637) sstableloader does not seem to load all the data present in the provided sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721268#comment-13721268 ] Vishy Kasar commented on CASSANDRA-5637: This turned out to be an issue with how we were reading back the data. The limit on how many columns to return on a read was set to 100 and it was not showing the newly added columns. Nothing more to do on this. sstableloader does not seem to load all the data present in the provided sstables - Key: CASSANDRA-5637 URL: https://issues.apache.org/jira/browse/CASSANDRA-5637 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.10 Reporter: Vishy Kasar Attachments: sstable-loader-log.txt I loaded about 1 million records in to my local cassandra. I then attempted to load these records using the sstableloader to prod cassandra using command sstableloader -v -u xyz -pw abc -d st11p02me-hpbb050130-i6 /tmp/prod_p02/containers When I read back the production data later, I do not find some rows that I inserted above. Few other rows are present though. Are there known bugs in sstableloader? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5637) sstableloader does not seem to load all the data present in the provided stables
Vishy Kasar created CASSANDRA-5637: -- Summary: sstableloader does not seem to load all the data present in the provided stables Key: CASSANDRA-5637 URL: https://issues.apache.org/jira/browse/CASSANDRA-5637 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.10 Reporter: Vishy Kasar Attachments: sstable-loader-log.txt I loaded about 1 million records in to my local cassandra. I then attempted to load these records using the sstableloader to prod cassandra using command sstableloader -v -u xyz -pw abc -d st11p02me-hpbb050130-i6 /tmp/prod_p02/containers When I read back the production data later, I do not find some rows that I inserted above. Few other rows are present though. Are there known bugs in sstableloader? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5637) sstableloader does not seem to load all the data present in the provided stables
[ https://issues.apache.org/jira/browse/CASSANDRA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-5637: --- Attachment: sstable-loader-log.txt Attached the verbose logs from sstableloader command. sstableloader does not seem to load all the data present in the provided stables Key: CASSANDRA-5637 URL: https://issues.apache.org/jira/browse/CASSANDRA-5637 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.10 Reporter: Vishy Kasar Attachments: sstable-loader-log.txt I loaded about 1 million records in to my local cassandra. I then attempted to load these records using the sstableloader to prod cassandra using command sstableloader -v -u xyz -pw abc -d st11p02me-hpbb050130-i6 /tmp/prod_p02/containers When I read back the production data later, I do not find some rows that I inserted above. Few other rows are present though. Are there known bugs in sstableloader? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5637) sstableloader does not seem to load all the data present in the provided sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishy Kasar updated CASSANDRA-5637: --- Summary: sstableloader does not seem to load all the data present in the provided sstables (was: sstableloader does not seem to load all the data present in the provided stables) sstableloader does not seem to load all the data present in the provided sstables - Key: CASSANDRA-5637 URL: https://issues.apache.org/jira/browse/CASSANDRA-5637 Project: Cassandra Issue Type: Bug Components: Tools Affects Versions: 1.1.10 Reporter: Vishy Kasar Attachments: sstable-loader-log.txt I loaded about 1 million records in to my local cassandra. I then attempted to load these records using the sstableloader to prod cassandra using command sstableloader -v -u xyz -pw abc -d st11p02me-hpbb050130-i6 /tmp/prod_p02/containers When I read back the production data later, I do not find some rows that I inserted above. Few other rows are present though. Are there known bugs in sstableloader? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
