Re: Tuning cassandra (compactions overall)

[Alain RODRIGUEZ]

I already had this kind of trouble while repairing a month ago. I have
problems that I am the only one to have. I guess I have something
wrong either in the configuration of my nodes or in my data that makes
them wrong after a restart/repair.

I am planning to try deploying an EC2 cluster with datastax AMI to be
sure of the integrity of the server. I will put my data into it and
try all maintenance reparation (repair, cleanup, rolling restart and
so on).

If you are interested in my start logs, here they are :

https://gist.github.com/2762493
https://gist.github.com/2762495

Thanks for your help Aaron.

Alain

2012/5/23 aaron morton :
> I've not heard of anything like that in the recent versions. There were some
> issues in the early
> 0.8 https://github.com/apache/cassandra/blob/trunk/NEWS.txt#L383
>
> If you are on a recent version can you please create a jira
> ticket https://issues.apache.org/jira/browse/CASSANDRA describing what you
> think happened.
>
> If you have kept the logs from the startup and can make them available
> please do.
>
> Thanks
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 23/05/2012, at 12:42 AM, Alain RODRIGUEZ wrote:
>
> "not sure what you mean by
> And after restarting the second one I have lost all the consistency of
> my data. All my statistics since September are totally false now in
> production
>
> Can you give some examples?"
>
> After restarting my 2 nodes (one after the other), All my counters
> have become wrong. The counters values were modified by the restart.
> Let's say I had a counter column called 20120101#click that value was
> 569, after the restart the value has become 751. I think that all the
> values have increased (I'm not sure) but all counters have increased
> in differents way, some values have increased a lot other just a bit.
>
> "Counter are not idempotent so if the client app retries TimedOut
> requests you can get an over count. That should not result in lost
> data."
>
> Some of these counters haven't be written since September and have
> still been modified by the restart.
>
> "Have you been running repair ?"
>
> Yes, Repair didn't helped. I have the feeling that repairing doesn't
> work on counters.
>
> I have restored the data now, but I am afraid of restarting any node.
> I can remain in this position too long...
>
>

Data Versioning Support

[Felipe Schmidt]

Doe's Cassandra support data versioning?

I'm trying to find it in many places but I'm not quite sure about it.

Regards,
Felipe Mathias Schmidt
(Computer Science UFRGS, RS, Brazil)

Re: how to get list of snapshots

[aaron morton]

> 1) ok, I can schedule snapshots using cron (snapshot's name will be ganarated 
> from current date)
>  how can I remove snapshots older than a week ? 
Delete the directory on disk. 

> 2) ok, I can enable increment backups. How can I remove incremental SSTables 
> older than 1 week ?
>  it's more tricky than with snapshots.
AFAIK delete the file on disk. 

> it will lead me to several find/cron/bash scripts. Single mistake and I can 
> delete cassandara data entirely.
Always work in the snapshots directory and you will be fine.

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 23/05/2012, at 10:45 PM, Илья Шипицин wrote:

> I seen that guide. It's missing several important things
> 
> 1) ok, I can schedule snapshots using cron (snapshot's name will be ganarated 
> from current date)
>  how can I remove snapshots older than a week ? 
> 
> 2) ok, I can enable increment backups. How can I remove incremental SSTables 
> older than 1 week ?
>  it's more tricky than with snapshots.
> 
> 
> it will lead me to several find/cron/bash scripts. Single mistake and I can 
> delete cassandara data entirely.
> 
> 2012/5/23 aaron morton 
>> 1) is there any good guide for scheduling backups ?
> this http://www.datastax.com/docs/1.0/operations/backup_restore ?
> 
>> 2) is there way to get list of snapshots ? (without "ls" in directory)
> No. 
> 
> Cheers
> 
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 23/05/2012, at 5:06 PM, Илья Шипицин wrote:
> 
>> Hello!
>> 
>> I'm about to schedule backups in the following way
>> 
>> a) snapshots are done daily
>> b) increment backups are enabled
>> 
>> so, backup will be consistent, very old snapshots must be removed (I guess, 
>> a week depth should be enough).
>> 
>> couple of questions:
>> 
>> 1) is there any good guide for scheduling backups ?
>> 2) is there way to get list of snapshots ? (without "ls" in directory)
>> 
>> Cheers,
>> Ilya Shipitsin
> 
> 

Re: Data Versioning Support

[R. Verlangen]

Hi Felipe,

There recently was a thread about (
http://www.mail-archive.com/user@cassandra.apache.org/msg22298.html  ). The
answer in short: no. However you can build your own data model to support
it.

Cheers!

2012/5/24 Felipe Schmidt 

> Doe's Cassandra support data versioning?
>
> I'm trying to find it in many places but I'm not quite sure about it.
>
> Regards,
> Felipe Mathias Schmidt
> (Computer Science UFRGS, RS, Brazil)
>



-- 
With kind regards,

Robin Verlangen
www.robinverlangen.nl

Re: Confusion regarding the terms "replica" and "replication factor"

[aaron morton]

This is partly historical. NTS (as it is now) has not always existed and was 
not always the default. In days gone by used to be a fella could run a mighty 
fine key-value store using just a Simple Replication Strategy. 

A different way to visualise it is a single ring with a Z axis for the DC's. 
When you look at the ring from the top you can see all the nodes. When you look 
at it from the side you can see the nodes are on levels that correspond to 
their DC. Simple Strategy looks at the ring from the top. NTS works through the 
layers of the ring. 

> If the hierarchy is Cluster ->
> DataCenter -> Node, why exactly do we need globally unique node tokens
> even though nodes are at the lowest level in the hierarchy.
Nodes having a DC is a feature of *some* snitches and utilised by the *some* of 
the replication strategies (and by the messaging system for network 
efficiency). For background, mapping from row tokens to nodes is based on 
http://en.wikipedia.org/wiki/Consistent_hashing

Hope that helps.
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 1:07 AM, java jalwa wrote:

> Thanks Aaron. That makes things clear.
> So I guess the 0 - 2^127 range for tokens corresponds to a cluster
> -level top-level ring. and then you add some logic on top of that with
> NTS to logically segment that range into sub-rings as per the notion
> of data clusters defined in NTS. Whats the advantage of having a
> single top-level ring ? intuitively it seems like each replication
> group could have a separate ring so that the same tokens can be
> assigned to nodes in different DC. If the hierarchy is Cluster ->
> DataCenter -> Node, why exactly do we need globally unique node tokens
> even though nodes are at the lowest level in the hierarchy.
> 
> Thanks again.
> 
> 
> On Wed, May 23, 2012 at 3:14 AM, aaron morton  wrote:
>>> Now if a row key hash is mapped to a range owned by a node in DC3,
>>> will the Node in DC3 still store the key as determined by the
>>> partitioner and then walk the ring and store 2 replicas each in DC1
>>> and DC2 ?
>> No, only nodes in the DC's specified in the NTS configuration will be 
>> replicas.
>> 
>>> Or will the co-ordinator node be aware of the
>>> replica placement strategy,
>>> and override the partitioner's decision and walk the ring until it
>>> first encounters a node in DC1 or DC2 ? and then place the remaining
>>> replicas ?
>> The NTS considers each DC to have it's own ring. This can make token 
>> selection in a multi DC environment confusing at times. There is something 
>> in the DS docs about it.
>> 
>> Cheers
>> 
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 23/05/2012, at 3:16 PM, java jalwa wrote:
>> 
>>> Hi all,
>>>  I am a bit confused regarding the terms "replica" and
>>> "replication factor". Assume that I am using RandomPartitioner and
>>> NetworkTopologyStrategy for replica placement.
>>> From what I understand, with a RandomPartitioner, a row key will
>>> always be hashed and be stored on the node that owns the range to
>>> which the key is mapped.
>>> http://www.datastax.com/docs/1.0/cluster_architecture/replication#networktopologystrategy.
>>> The example here, talks about having 2 data centers and a replication
>>> factor of 4 with 2 replicas in each datacenter, so the strategy is
>>> configured as DC1:2 and DC2:2. Now suppose I add another datacenter
>>> DC3, and do not change the NetworkTopologyStrategy.
>>> Now if a row key hash is mapped to a range owned by a node in DC3,
>>> will the Node in DC3 still store the key as determined by the
>>> partitioner and then walk the ring and store 2 replicas each in DC1
>>> and DC2 ? Will that mean that I will then have 5 replicas in the
>>> cluster and not 4 ? Or will the co-ordinator node be aware of the
>>> replica placement strategy,
>>> and override the partitioner's decision and walk the ring until it
>>> first encounters a node in DC1 or DC2 ? and then place the remaining
>>> replicas ?
>>> 
>>> Thanks.
>> 

Re: Retrieving old data version for a given row

[Felipe Schmidt]

Ok... it's really strange to me that Cassandra doesn't support data
versioning cause all of other key-value databases support it (at least
those who I know).

I have one remaining question:
-in the case that I have more than 1 SSTable in the disk for the same
column but with different data versions, is it possible to make a
query to get the old version instead of the newest one?

Regards,
Felipe Mathias Schmidt
(Computer Science UFRGS, RS, Brazil)




2012/5/16 Dave Brosius :
> You're in for a world of hurt going down that rabbit hole. If you truely
> want version data then you should think about changing your keying to
> perhaps be a composite key where key is of form
>
> NaturalKey/VersionId
>
> Or if you want the versioning at the column level, use composite columns
> with ColumnName/VersionId format
>
>
>
>
> On 05/16/2012 10:16 AM, Felipe Schmidt wrote:
>>
>> That was very helpfull, thank you very much!
>>
>> I still have some questions:
>> -it is possible to make Cassandra keep old value data after flushing?
>> The same question for the memTable, before flushing. Seems to me that
>> when I update some tuple, the old data will be overwrited in memTable,
>> even before flushing.
>> -it is possible to scan values from the memtable, maybe using the
>> so-called Thrift API? Using the client-api I can just see the newest
>> data version, I can't see what's really happening with the memTable.
>>
>> I ask that cause what I'll try to do is a Change Data Capture to
>> Cassandra and the answers will define what kind of aproaches I'm able
>> to use.
>>
>> Thanks in advance.
>>
>> Regards,
>> Felipe Mathias Schmidt
>> (Computer Science UFRGS, RS, Brazil)
>>
>>
>> 2012/5/14 aaron morton:
>>>
>>> Cassandra does not provide access to multiple versions of the same
>>> column.
>>> It is essentially implementation detail.
>>>
>>> All mutations are written to the commit log in a binary format, see the
>>> o.a.c.db.RowMutation.getSerializedBuffer() (If you want to tail it for
>>> analysis you may want to change commitlog_sync in cassandra.yaml)
>>>
>>> Here is post about looking at multiple versions columns in an
>>> sstable http://thelastpickle.com/2011/05/15/Deletes-and-Tombstones/
>>>
>>> Remember that not all "versions" of a column are written to disk
>>>  (see http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/).
>>> Also
>>> compaction will compress multiple versions of the same column from
>>> multiple
>>> files into a single version in a single file .
>>>
>>> Hope that helps.
>>>
>>>
>>> -
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 14/05/2012, at 9:50 PM, Felipe Schmidt wrote:
>>>
>>> Yes, I need this information just for academic purposes.
>>>
>>> So, to read old data values, I tried to open the Commitlog using tail
>>> -f and also the log files viewer of Ubuntu, but I can not see many
>>> informations inside of the log!
>>> Is there any other way to open this log? I didn't find any Cassandra
>>> API for this purpose.
>>>
>>> Thanks averybody in advance.
>>>
>>> Regards,
>>> Felipe Mathias Schmidt
>>> (Computer Science UFRGS, RS, Brazil)
>>>
>>>
>>>
>>>
>>> 2012/5/14 zhangcheng2:
>>>
>>> After compaciton, the old version data will gone!
>>>
>>>
>>> 
>>>
>>> zhangcheng2
>>>
>>>
>>> From: Felipe Schmidt
>>>
>>> Date: 2012-05-14 05:33
>>>
>>> To: user
>>>
>>> Subject: Retrieving old data version for a given row
>>>
>>> I'm trying to retrieve old data version for some row but it seems not
>>>
>>> be possible. I'm a beginner  with Cassandra and the unique aproach I
>>>
>>> know is looking to the SSTable in the storage folder, but if I insert
>>>
>>> some column and right after insert another value to the same row,
>>>
>>> after flushing, I only get the last value.
>>>
>>> Is there any way to get the old data version? Obviously, before
>>> compaction.
>>>
>>>
>>> Regards,
>>>
>>> Felipe Mathias Schmidt
>>>
>>> (Computer Science UFRGS, RS, Brazil)
>>>
>>>
>>>
>

Re: Error loading data: Internal error processing get_range_slices / Unavailable Exception

[aaron morton]

I'm not sure what those log messages are from. But….

> UnknownException: [host=192.168.2.13(192.168.2.13):9160, latency=11(31), 
> attempts=1] SchemaDisagreementException()

Sounds a bit like. 

http://wiki.apache.org/cassandra/FAQ#schema_disagreement

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 4:15 PM, Abhijit Chanda wrote:

> Hi All,
> i am facing problem while setting up my database. The error under mentioned 
> is reflected every time i try to
> setup the DB. Unable to understand why these are occurring?  though 
> previously it was working fine, i guess
> it is some connection related issues. 
> 
> UnknownException: [host=192.168.2.13(192.168.2.13):9160, latency=11(31), 
> attempts=1] SchemaDisagreementException()
> NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0] 
> No hosts to borrow from
> Describe Keyspace: vCRIME
> OperationTimeoutException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=10002(10002), attempts=1] TimedOutException()
> OperationTimeoutException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=10001(10001), attempts=1] TimedOutException()
> OperationTimeoutException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=1(1), attempts=1] TimedOutException()
> OperationTimeoutException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=10001(10001), attempts=1] TimedOutException()
> OperationTimeoutException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=10001(10001), attempts=1] TimedOutException()
> OperationTimeoutException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=10001(10001), attempts=1] TimedOutException()
> TokenRangeOfflineException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=2(2), attempts=1] UnavailableException()
> TokenRangeOfflineException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=2(2), attempts=1] UnavailableException()
> TokenRangeOfflineException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=2(2), attempts=1] UnavailableException()
> TokenRangeOfflineException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=2(2), attempts=1] UnavailableException()
> TokenRangeOfflineException: [host=192.168.2.13(192.168.2.13):9160, 
> latency=2(2), attempts=1] UnavailableException()
> 
> Regards,
> Abhijit

Re: Replication factor

[aaron morton]

ReadRepair means including all UP replicas in the request, waiting 
asynchronously after the read has completed, resolving and repairing 
differences. If you read at QUOURM with RR running, ALL (replace) nodes will 
perform a read. 

At any CL > ONE the responses from CL nodes are reconciled and differences are 
repaired using a mechanism similar to RR. This has to happen before the 
response can be sent to the client.

The naming does not help, but they are a different. RR is a background process 
designed to reduce the chance of an inconsistent read. It's needed less since 
changed to Hinted Handoff in 1.0.

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 5:47 AM, Daning Wang wrote:

> Thanks guys.
> 
> Aaron, I am confused about this. from wiki 
> http://wiki.apache.org/cassandra/ReadRepair, looks for any consistency level. 
> Read Repair will be done either before or after responding data.
> 
>   >Read Repair does not run at CL ONE
> 
> Daning
> 
> On Wed, May 23, 2012 at 3:51 AM, Viktor Jevdokimov 
>  wrote:
> > When RF == number of nodes, and you read at CL ONE you will always be 
> > reading locally.
> 
> “always be reading locally” – only if Dynamic Snitch is “off”. With dynamic 
> snitch “on” request may be redirected to other node, which may introduce 
> latency spikes.
> 
>  
> 
>  
> 
> 
> 
> Best regards / Pagarbiai
> Viktor Jevdokimov
> Senior Developer
> 
> Email: viktor.jevdoki...@adform.com
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider
> What is Adform: watch this short video
>  
> 
> Disclaimer: The information contained in this message and attachments is 
> intended solely for the attention and use of the named addressee and may be 
> confidential. If you are not the intended recipient, you are reminded that 
> the information remains the property of the sender. You must not use, 
> disclose, distribute, copy, print or rely on this e-mail. If you have 
> received this message in error, please contact the sender immediately and 
> irrevocably delete this message and any copies.
> 
> From: aaron morton [mailto:aa...@thelastpickle.com] 
> Sent: Wednesday, May 23, 2012 13:00
> To: user@cassandra.apache.org
> Subject: Re: Replication factor
> 
>  
> 
> RF is normally adjusted to modify availability (see 
> http://thelastpickle.com/2011/06/13/Down-For-Me/)
> 
>  
> 
> for example, if I have 4 nodes cluster in one data center, how can RF=2 vs 
> RF=4 affect read performance? If consistency level is ONE, looks reading does 
> not need to go to another hop to get data if RF=4, but it would do more work 
> on read repair in the background.
> 
> Read Repair does not run at CL ONE.
> 
> When RF == number of nodes, and you read at CL ONE you will always be reading 
> locally. But with a low consistency.
> 
> If you read with QUORUM when RF == number of nodes you will still get some 
> performance benefit from the data being read locally.
> 
>  
> 
> Cheers
> 
>  
> 
>  
> 
> -
> 
> Aaron Morton
> 
> Freelance Developer
> 
> @aaronmorton
> 
> http://www.thelastpickle.com
> 
>  
> 
> On 23/05/2012, at 9:34 AM, Daning Wang wrote:
> 
> 
> 
> 
> Hello,
> 
> What is the pros and cons to choose different number of replication factor in 
> term of performance? if space is not a concern.
> 
> for example, if I have 4 nodes cluster in one data center, how can RF=2 vs 
> RF=4 affect read performance? If consistency level is ONE, looks reading does 
> not need to go to another hop to get data if RF=4, but it would do more work 
> on read repair in the background.
> 
> Can you share some insights about this?
> 
> Thanks in advance,
> 
> Daning
> 
>  
> 
> 

Re: Replication factor

[aaron morton]

Your experience is when using CL ONE the Dynamic Snitch is moving local reads 
off to other nodes and this is causing spikes in read latency ? 

Did you notice what was happening on the node for the DS to think it was so 
slow ? Was compaction or repair going on ? 
 
Have you played with the badness threshold 
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L472 ? 

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 5:28 PM, Viktor Jevdokimov wrote:

> Depends on use case. For ours we have another experience and statistics, when 
> turning dynamic snitch off makes overall latency and spikes much, much lower.
>  
>  
> 
> 
> Best regards / Pagarbiai
> Viktor Jevdokimov
> Senior Developer
> 
> Email: viktor.jevdoki...@adform.com
> Phone: +370 5 212 3063, Fax +370 5 261 0453
> J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
> Follow us on Twitter: @adforminsider
> What is Adform: watch this short video
> 
> 
> Disclaimer: The information contained in this message and attachments is 
> intended solely for the attention and use of the named addressee and may be 
> confidential. If you are not the intended recipient, you are reminded that 
> the information remains the property of the sender. You must not use, 
> disclose, distribute, copy, print or rely on this e-mail. If you have 
> received this message in error, please contact the sender immediately and 
> irrevocably delete this message and any copies.
> 
> From: Brandon Williams [mailto:dri...@gmail.com] 
> Sent: Thursday, May 24, 2012 02:35
> To: user@cassandra.apache.org
> Subject: Re: Replication factor
>  
> On Wed, May 23, 2012 at 5:51 AM, Viktor Jevdokimov 
>  wrote:
> > When RF == number of nodes, and you read at CL ONE you will always be 
> > reading locally.
> “always be reading locally” – only if Dynamic Snitch is “off”. With dynamic 
> snitch “on” request may be redirected to other node, which may introduce 
> latency spikes.
>  
> Actually it's preventing spikes, since if it won't read locally that means 
> the local replica is in worse shape than the rest (compacting, repairing, 
> etc.)
>  
> -Brandon 

Re: unknown exception with hector

[aaron morton]

Dropped read messages occur when the node could not process a read task within 
rpc_timeout. It generally means the cluster has been overwhelmed at some point: 
 too many requests, to much GC, compaction hurting, etc. 

Check the server side logs for errors but I doubt it is related to the call 
stack below. 

Have you confirmed that you are using framed transport on the client?

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 5:52 PM, Deno Vichas wrote:

> i've notice the my nodes seem to have a large (?, not really sure what 
> acceptable numbers are) read dropped count from tpstats.  could they be 
> related? 
> 
> On 5/23/2012 2:55 AM, aaron morton wrote:
>> 
>> No sure but 
>> 
>>>at 
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> 
>> Looks like the client is not using framed transport. The server defaults to 
>> framed.
>> 
>> Cheers
>> 
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 23/05/2012, at 5:35 AM, Deno Vichas wrote:
>> 
>>> could somebody clue me in to the cause of this exception?  i see these 
>>> randomly.
>>> 
>>> AnalyzerService-2 2012-05-22 13:28:00,385 :: WARN  
>>> cassandra.connection.HConnectionManager  - Exception:
>>> me.prettyprint.hector.api.exceptions.HectorTransportException: 
>>> org.apache.thrift.transport.TTransportException
>>>at 
>>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:39)
>>>at 
>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:851)
>>>at 
>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:840)
>>>at 
>>> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:99)
>>>at 
>>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:243)
>>>at 
>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)
>>>at 
>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl.getColumn(KeyspaceServiceImpl.java:857)
>>>at 
>>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:57)
>>>at 
>>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:52)
>>>at 
>>> me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
>>>at 
>>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
>>>at 
>>> me.prettyprint.cassandra.model.thrift.ThriftColumnQuery.execute(ThriftColumnQuery.java:51)
>>>at 
>>> com.stocktouch.dao.StockDaoImpl.getHistorical(StockDaoImpl.java:365)
>>>at 
>>> com.stocktouch.dao.StockDaoImpl.getHistoricalQuote(StockDaoImpl.java:433)
>>>at 
>>> com.stocktouch.service.StockHistoryServiceImpl.getHistoricalQuote(StockHistoryServiceImpl.java:480)
>>>at 
>>> com.stocktouch.service.AnalyzerServiceImpl.getClose(AnalyzerServiceImpl.java:180)
>>>at 
>>> com.stocktouch.service.AnalyzerServiceImpl.calcClosingPrices(AnalyzerServiceImpl.java:90)
>>>at 
>>> com.stocktouch.service.AnalyzerServiceImpl.nightlyRollup(AnalyzerServiceImpl.java:66)
>>>at 
>>> com.stocktouch.service.AnalyzerServiceImpl$2.run(AnalyzerServiceImpl.java:55)
>>>at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>at 
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>at java.lang.Thread.run(Thread.java:662)
>>> Caused by: org.apache.thrift.transport.TTransportException
>>>at 
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>at 
>>> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>at 
>>> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>at 
>>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>at 
>>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>at 
>>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>at 
>>> org.apache.cassandra.thrift.Cassandra$Client.recv_get(Cassandra.java:509)
>>>at 
>>> org.apache.cassandra.thrift.Cassandra$Client.get(Cassandra.java:492)
>>>at 
>>> me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:846)
>>>... 20 more
>>> 
>>> 
>>> thanks,
>>> deno
>>> 
>>

Re: Retrieving old data version for a given row

[aaron morton]

> Ok... it's really strange to me that Cassandra doesn't support data
> versioning cause all of other key-value databases support it (at least
> those who I know).
You can design it into your data model if you need it.
 
> I have one remaining question:
> -in the case that I have more than 1 SSTable in the disk for the same
> column but with different data versions, is it possible to make a
> query to get the old version instead of the newest one?
No.
There is only ever 1 value for a column. 
The "older" copies of the column in the SSTables are artefacts of immutable on 
disk structures. 
If you want to see what's inside an SSTable use bin/sstable2json

Cheers
  
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 9:42 PM, Felipe Schmidt wrote:

> Ok... it's really strange to me that Cassandra doesn't support data
> versioning cause all of other key-value databases support it (at least
> those who I know).
> 
> I have one remaining question:
> -in the case that I have more than 1 SSTable in the disk for the same
> column but with different data versions, is it possible to make a
> query to get the old version instead of the newest one?
> 
> Regards,
> Felipe Mathias Schmidt
> (Computer Science UFRGS, RS, Brazil)
> 
> 
> 
> 
> 2012/5/16 Dave Brosius :
>> You're in for a world of hurt going down that rabbit hole. If you truely
>> want version data then you should think about changing your keying to
>> perhaps be a composite key where key is of form
>> 
>> NaturalKey/VersionId
>> 
>> Or if you want the versioning at the column level, use composite columns
>> with ColumnName/VersionId format
>> 
>> 
>> 
>> 
>> On 05/16/2012 10:16 AM, Felipe Schmidt wrote:
>>> 
>>> That was very helpfull, thank you very much!
>>> 
>>> I still have some questions:
>>> -it is possible to make Cassandra keep old value data after flushing?
>>> The same question for the memTable, before flushing. Seems to me that
>>> when I update some tuple, the old data will be overwrited in memTable,
>>> even before flushing.
>>> -it is possible to scan values from the memtable, maybe using the
>>> so-called Thrift API? Using the client-api I can just see the newest
>>> data version, I can't see what's really happening with the memTable.
>>> 
>>> I ask that cause what I'll try to do is a Change Data Capture to
>>> Cassandra and the answers will define what kind of aproaches I'm able
>>> to use.
>>> 
>>> Thanks in advance.
>>> 
>>> Regards,
>>> Felipe Mathias Schmidt
>>> (Computer Science UFRGS, RS, Brazil)
>>> 
>>> 
>>> 2012/5/14 aaron morton:
 
 Cassandra does not provide access to multiple versions of the same
 column.
 It is essentially implementation detail.
 
 All mutations are written to the commit log in a binary format, see the
 o.a.c.db.RowMutation.getSerializedBuffer() (If you want to tail it for
 analysis you may want to change commitlog_sync in cassandra.yaml)
 
 Here is post about looking at multiple versions columns in an
 sstable http://thelastpickle.com/2011/05/15/Deletes-and-Tombstones/
 
 Remember that not all "versions" of a column are written to disk
  (see http://thelastpickle.com/2011/04/28/Forces-of-Write-and-Read/).
 Also
 compaction will compress multiple versions of the same column from
 multiple
 files into a single version in a single file .
 
 Hope that helps.
 
 
 -
 Aaron Morton
 Freelance Developer
 @aaronmorton
 http://www.thelastpickle.com
 
 On 14/05/2012, at 9:50 PM, Felipe Schmidt wrote:
 
 Yes, I need this information just for academic purposes.
 
 So, to read old data values, I tried to open the Commitlog using tail
 -f and also the log files viewer of Ubuntu, but I can not see many
 informations inside of the log!
 Is there any other way to open this log? I didn't find any Cassandra
 API for this purpose.
 
 Thanks averybody in advance.
 
 Regards,
 Felipe Mathias Schmidt
 (Computer Science UFRGS, RS, Brazil)
 
 
 
 
 2012/5/14 zhangcheng2:
 
 After compaciton, the old version data will gone!
 
 
 
 
 zhangcheng2
 
 
 From: Felipe Schmidt
 
 Date: 2012-05-14 05:33
 
 To: user
 
 Subject: Retrieving old data version for a given row
 
 I'm trying to retrieve old data version for some row but it seems not
 
 be possible. I'm a beginner  with Cassandra and the unique aproach I
 
 know is looking to the SSTable in the storage folder, but if I insert
 
 some column and right after insert another value to the same row,
 
 after flushing, I only get the last value.
 
 Is there any way to get the old data version? Obviously, before
 compaction.
 
 
 Regards,
 
 Felipe Mathias Schmidt

RE: Replication factor

[Viktor Jevdokimov]

All data is in the page cache. No repairs. Compactions not hitting disk for 
read. CPU <50%. ParNew GC <100 ms in average.

After one compaction completes, new sstable is not in page cache, there may be 
a disk usage spike before data is cached, so local reads gets slower for a 
moment, comparing with other nodes. Redirecting almost all requests to other 
nodes finally ends up with a huge latency spike almost on all nodes, especially 
when ParNew GC may spike on one node (>200ms). We call it "cluster hiccup", 
when incoming and outgoing network traffic drops for a moment.

And such hiccups happens several times an hour, few seconds long. Playing with 
badness threshold did not gave a lot better results, but turning DS off 
completely fixed all problems with latencies, node spikes, cluster hiccups and 
network traffic drops.

In our case, our client is selecting endpoints for a key by calculating a 
token, so we always hit a replica.




Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer

Email: viktor.jevdoki...@adform.com
Phone: +370 5 212 3063, Fax +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
Follow us on Twitter: @adforminsider
What is Adform: watch this short video

[Adform News] 


Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.

From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: Thursday, May 24, 2012 13:00
To: user@cassandra.apache.org
Subject: Re: Replication factor

Your experience is when using CL ONE the Dynamic Snitch is moving local reads 
off to other nodes and this is causing spikes in read latency ?

Did you notice what was happening on the node for the DS to think it was so 
slow ? Was compaction or repair going on ?

Have you played with the badness threshold 
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L472 ?

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 5:28 PM, Viktor Jevdokimov wrote:


Depends on use case. For ours we have another experience and statistics, when 
turning dynamic snitch off makes overall latency and spikes much, much lower.



Best regards / Pagarbiai
Viktor Jevdokimov
Senior Developer

Email: viktor.jevdoki...@adform.com
Phone: +370 5 212 3063, Fax +370 5 261 0453
J. Jasinskio 16C, LT-01112 Vilnius, Lithuania
Follow us on Twitter: @adforminsider
What is Adform: watch this short video




Disclaimer: The information contained in this message and attachments is 
intended solely for the attention and use of the named addressee and may be 
confidential. If you are not the intended recipient, you are reminded that the 
information remains the property of the sender. You must not use, disclose, 
distribute, copy, print or rely on this e-mail. If you have received this 
message in error, please contact the sender immediately and irrevocably delete 
this message and any copies.

From: Brandon Williams 
[mailto:dri...@gmail.com]
Sent: Thursday, May 24, 2012 02:35
To: user@cassandra.apache.org
Subject: Re: Replication factor

On Wed, May 23, 2012 at 5:51 AM, Viktor Jevdokimov 
mailto:viktor.jevdoki...@adform.com>> wrote:
> When RF == number of nodes, and you read at CL ONE you will always be reading 
> locally.
"always be reading locally" - only if Dynamic Snitch is "off". With dynamic 
snitch "on" request may be redirected to other node, which may introduce 
latency spikes.

Actually it's preventing spikes, since if it won't read locally that means the 
local replica is in worse shape than the rest (compacting, repairing, etc.)

-Brandon

<>

what about an "hybrid" partitioner for CF with composite row key ?

[DE VITO Dominique]

Hi,

We have defined a CF with a composite row key that sounds like (folder id, doc 
id).

For our app, one very common pattern is accessing, through one ui action, some 
bunch of data with the following row keys: (id, id_1), (id, id_2), (id, id_3)...
So, multiple rows are accessed, but all row keys have the same 1st part "folder 
id".

* with the BOP: for one ui action, one simple node is requested (in average), 
but it's much harder to balances the cluster nodes
* with the RP: for one ui action, many nodes may be requested, but it's simpler 
to balance the cluster

one sweeter(?) partitioner would be a partitioner that would distribute a row 
according only to the first part of its key (= according to "folder id" only).

Is it doable to implement such a partitioner ?
Thanks.

Regards,
Dominique

Re: Query on how to count the total number of rowkeys and columns in them

[Віталій Тимчишин]

You should read multiple "batches" specifying last key received from
previous batch as first key for next one.
For large databases I'd recommend you to use statistical approach (if it's
feasible). With random parittioner it works well.
Don't read the whole db. Knowing whole keyspace you can read part, get
number of records per key (<1), then multiply by keyspace size and get your
total.
You can even implement an algorithm that will work until required precision
is obtained (simply after each batch compare you previous and current
estimate).
For me it's enough to read ~1% of DB to get good result.

Best regards, Vitalii Tymchyshyn

2012/5/24 Prakrati Agrawal 

>  Hi
>
> ** **
>
> I am trying to learn Cassandra and I have one doubt. I am using the Thrift
> API, to count the number of row keys I am using KeyRange to specify the row
> keys. To count all of them, I specify the start and end as “new byte[0]”.
> But the count is set to 100 by default. How do I use this method to count
> the keys if I don’t know the actual number of keys in my Cassandra
> database? Please help me
>
>  **
>
-- 
Best regards,
 Vitalii Tymchyshyn

Re: inconsistent snapshots

[Vijay]

Can you attach the cassandra logs? How was it restored? Priam chunks and
compresses before uploading it to S3 and when you restore you have to go
thought Priam to restore...

Regards,




On Thu, May 24, 2012 at 3:58 AM, Jose Flexa  wrote:

> Hi all,
>
> Having issues with cassandra 1.0.8 on AWS EC2 and Netflix Priam with a
> XFS FS: seems snapshots are broken.
> When I restore then and run nodetool repair cassandra cant load them
> complaning
> . Anybody had a similiar experience or any suggestions?
>
> Thanks in advance
> José
>

Migrating from a windows cluster to a linux cluster.

[Henrik Schröder]

Hey everyone,

We're trying to migrate a cassandra cluster from a bunch of Windows
machines to a bunch of (newer and more powerful) Linux machines.

Our initial plan was to simply bootstrap the Linux servers into the cluster
one by one, and then decommission the old servers one by one. However, when
we try to join a Linux server to the cluster, we get the following error:

ERROR 11:52:22,959 Fatal exception in thread Thread[Thread-21,5,main]
java.lang.AssertionError: Filename must include parent directory.
at
org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:148)
at
org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.deserialize(PendingFile.java:138)
at
org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.deserialize(StreamHeader.java:88)
at
org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.deserialize(StreamHeader.java:70)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:80)

A quick googling reveals that the cause is the simple fact that Cassandra
is transmitting the full path of the datafiles with the native directory
separator, "\", and the Linux servers expect it to be "/", and get confused
as a result.

We're running version 1.0.8. Is this fixed in a later release? Will this be
fixed in a later release?

Are there any other ways of doing the migration? What happens if we join
the new servers without bootstrapping and run repair? Are there any other
ugly hacks or workaround we can do? We're not looking to run a mixed
cluster, we just want to migrate all the data as painlessly as possible.


/Henrik

Composite keys question

[Roland Mechler]

Suppose I have a table in CQL3 with a 2 part composite, and I do a select
that specifies just the second part of the key (not the partition key),
will this result in a full table scan, or is the second part of the key
indexed?

Example:

cqlsh:"Keyspace1"> CREATE TABLE test_table (part1 text, part2 text, data
text, PRIMARY KEY (part1, part2));
cqlsh:"Keyspace1"> INSERT INTO test_table (part1, part2, data) VALUES
('1','1','a');
cqlsh:"Keyspace1"> SELECT * FROM test_table WHERE part2 = '1';
 part1 | part2 | data
---+---+--
 1 | 1 |a

-Roland

Re: Migrating from a windows cluster to a linux cluster.

[Henrik Schröder]

Hey, we thought a bit about it and came up with another solution:

We shut down Cassandra on one of the windows servers, copy over the data
directory to one of the Linux servers, delete the LocationInfo files from
the system keyspace, and start it up.

It should read the saved token from the datafiles, it should have all the
data associated with that token, and on joining the cluster it should just
pop in at the right place, but with a new ip address. And then we repeat
that for each server.

Will this work? Or is there a better way?


/Henrik

On Thu, May 24, 2012 at 7:41 PM, Henrik Schröder  wrote:

> Hey everyone,
>
> We're trying to migrate a cassandra cluster from a bunch of Windows
> machines to a bunch of (newer and more powerful) Linux machines.
>
> Our initial plan was to simply bootstrap the Linux servers into the
> cluster one by one, and then decommission the old servers one by one.
> However, when we try to join a Linux server to the cluster, we get the
> following error:
>
> ERROR 11:52:22,959 Fatal exception in thread Thread[Thread-21,5,main]
> java.lang.AssertionError: Filename must include parent directory.
> at
> org.apache.cassandra.io.sstable.Descriptor.fromFilename(Descriptor.java:148)
> at
> org.apache.cassandra.streaming.PendingFile$PendingFileSerializer.deserialize(PendingFile.java:138)
> at
> org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.deserialize(StreamHeader.java:88)
> at
> org.apache.cassandra.streaming.StreamHeader$StreamHeaderSerializer.deserialize(StreamHeader.java:70)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:80)
>
> A quick googling reveals that the cause is the simple fact that Cassandra
> is transmitting the full path of the datafiles with the native directory
> separator, "\", and the Linux servers expect it to be "/", and get confused
> as a result.
>
> We're running version 1.0.8. Is this fixed in a later release? Will this
> be fixed in a later release?
>
> Are there any other ways of doing the migration? What happens if we join
> the new servers without bootstrapping and run repair? Are there any other
> ugly hacks or workaround we can do? We're not looking to run a mixed
> cluster, we just want to migrate all the data as painlessly as possible.
>
>
> /Henrik
>

Re: Migrating from a windows cluster to a linux cluster.

[Brandon Williams]

On Thu, May 24, 2012 at 12:41 PM, Henrik Schröder  wrote:
> We're running version 1.0.8. Is this fixed in a later release? Will this be
> fixed in a later release?

No, mixed-OS clusters are unsupported.

> Are there any other ways of doing the migration? What happens if we join the
> new servers without bootstrapping and run repair? Are there any other ugly
> hacks or workaround we can do? We're not looking to run a mixed cluster, we
> just want to migrate all the data as painlessly as possible.

Start the linux cluster independently and use sstableloader from the
windows cluster to populate it.

-Brandon

Why does a large compaction on one node affect the entire cluster?

[Thomas van Neerijnen]

Hi all

I am running Cassandra 1.0.10 installed from the apache debs on ubuntu
11.10 on a 7 node cluster.

I moved some tokens around my cluster and now have one node compacting a
large Leveled compaction column family. It has done about 5k out of 10k
outstanding compactions today. The other nodes have all finished.

The weird thing is when it hits a big-ish chunk to compact, for example:
pending tasks: 4555
  compaction typekeyspace   column family bytes
compacted bytes total  progress
   Compaction  PlayerPlayerDetail
213097  4286616517 0.00%
, I see heap usage on it AND all other nodes go insane.
Normal operation on all nodes is a leisurely saw-toothed climb to a CMS at
just below 3/4 heap size every 10 minutes or so.
During the big-ish compaction all nodes in the cluster CMS multiple times
in a minute, with the peaks getting close to heap size.

So my question is why does one node compacting put so much memory pressure
on all the other nodes in the cluster and ruin my day?

Re: Migrating from a windows cluster to a linux cluster.

[Henrik Schröder]

On Thu, May 24, 2012 at 8:07 PM, Brandon Williams  wrote:

> > Are there any other ways of doing the migration? What happens if we join
> the
> > new servers without bootstrapping and run repair? Are there any other
> ugly
> > hacks or workaround we can do? We're not looking to run a mixed cluster,
> we
> > just want to migrate all the data as painlessly as possible.
>
> Start the linux cluster independently and use sstableloader from the
> windows cluster to populate it.
>
>
Ok. It's important for us to not have any downtime, so how about this
solution:

We startup the Linux cluster independently.
We configure our application to send all Cassandra writes to both clusters,
but only read from the Windows cluster.
We run sstableloader on each windows server (Is it possible to do in
parallell?), sending whatever it has to the Linux cluster.
When it's done on all Windows servers, we configure our application to only
talk to the Linux cluster.

The only issue with this is the timestamps of the data and tombstones in
each sstable, will they be preserved by sstableloader? What about deletes
of non-existing keys? Will they be stored in the Linux cluster so that when
sstableloader inserts the key later, it's resolved as being deleted?


/Henrik

Re: Migrating from a windows cluster to a linux cluster.

[Brandon Williams]

On Thu, May 24, 2012 at 1:50 PM, Henrik Schröder  wrote:
> Ok. It's important for us to not have any downtime, so how about this
> solution:
>
> We startup the Linux cluster independently.
> We configure our application to send all Cassandra writes to both clusters,
> but only read from the Windows cluster.
> We run sstableloader on each windows server (Is it possible to do in
> parallell?), sending whatever it has to the Linux cluster.
> When it's done on all Windows servers, we configure our application to only
> talk to the Linux cluster.

That sounds fine, with the caveat that you can't run sstableloader
from a machine running Cassandra before 1.1, so copying the sstables
manually (assuming both clusters are the same size and have the same
tokens) might be better.

> The only issue with this is the timestamps of the data and tombstones in
> each sstable, will they be preserved by sstableloader? What about deletes of
> non-existing keys? Will they be stored in the Linux cluster so that when
> sstableloader inserts the key later, it's resolved as being deleted?

None of that should be a problem.

-Brandon

Re: Migrating from a windows cluster to a linux cluster.

[Steve Neely]

It also seems like a dark deployment of your new cluster is a great method
for testing the Linux-based systems *before* switching your mision critical
traffic over. Monitor them for a while with real traffic and you can have
confidence that they'll function correctly when you perform the switchover.

-- Steve


On Thu, May 24, 2012 at 1:28 PM, Brandon Williams  wrote:

> On Thu, May 24, 2012 at 1:50 PM, Henrik Schröder 
> wrote:
> > Ok. It's important for us to not have any downtime, so how about this
> > solution:
> >
> > We startup the Linux cluster independently.
> > We configure our application to send all Cassandra writes to both
> clusters,
> > but only read from the Windows cluster.
> > We run sstableloader on each windows server (Is it possible to do in
> > parallell?), sending whatever it has to the Linux cluster.
> > When it's done on all Windows servers, we configure our application to
> only
> > talk to the Linux cluster.
>
> That sounds fine, with the caveat that you can't run sstableloader
> from a machine running Cassandra before 1.1, so copying the sstables
> manually (assuming both clusters are the same size and have the same
> tokens) might be better.
>
> > The only issue with this is the timestamps of the data and tombstones in
> > each sstable, will they be preserved by sstableloader? What about
> deletes of
> > non-existing keys? Will they be stored in the Linux cluster so that when
> > sstableloader inserts the key later, it's resolved as being deleted?
>
> None of that should be a problem.
>
> -Brandon
>

Re: Migrating from a windows cluster to a linux cluster.

[Henrik Schröder]

On Thu, May 24, 2012 at 9:28 PM, Brandon Williams  wrote:

>
> That sounds fine, with the caveat that you can't run sstableloader
> from a machine running Cassandra before 1.1, so copying the sstables
> manually (assuming both clusters are the same size and have the same
> tokens) might be better.
>

Why is version 1.1 required for sstableloader? We're running 1.0.x on both
clusters, but we can of course upgrade if that's required.


> > The only issue with this is the timestamps of the data and tombstones in
> > each sstable, will they be preserved by sstableloader? What about
> deletes of
> > non-existing keys? Will they be stored in the Linux cluster so that when
> > sstableloader inserts the key later, it's resolved as being deleted?
>
> None of that should be a problem.
>

Excellent, thanks!


/Henrik

Re: Migrating from a windows cluster to a linux cluster.

[Brandon Williams]

On Thu, May 24, 2012 at 3:36 PM, Henrik Schröder  wrote:
>> That sounds fine, with the caveat that you can't run sstableloader
>> from a machine running Cassandra before 1.1, so copying the sstables
>> manually (assuming both clusters are the same size and have the same
>> tokens) might be better.
>
>
> Why is version 1.1 required for sstableloader? We're running 1.0.x on both
> clusters, but we can of course upgrade if that's required.

Before 1.1 sstableloader is a fat client, and thus can't coexist with
an existing Cassandra instance on the same machine.

-Brandon

Re: Migrating from a windows cluster to a linux cluster.

[Rob Coli]

On Thu, May 24, 2012 at 12:44 PM, Steve Neely  wrote:
> It also seems like a dark deployment of your new cluster is a great method
> for testing the Linux-based systems before switching your mision critical
> traffic over. Monitor them for a while with real traffic and you can have
> confidence that they'll function correctly when you perform the switchover.

FWIW, I would love to see graphs which show their compared performance
under identical write load and then show the cut-over point for reads
between the two clusters. My hypothesis is that your linux cluster
will magically be much more perfomant/less loaded due to many
linux-specific optimizations in Cassandra, but I'd dig seeing this
illustrated in an apples to apples sense with real app traffic.

=Rob

-- 
=Robert Coli
AIM>ALK - rc...@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb

Re: Counters and replication factor

[Edward Capriolo]

Also it does not sound like you have run anti entropy repair. You should do
that when upping rf.
On Monday, May 21, 2012, Radim Kolar  wrote:
> Dne 26.3.2012 19:17, aaron morton napsal(a):
>>
>> Can you describe the situations where counter updates are lost or go
backwards ?
>>
>> Do you ever get TimedOutExceptions when performing counter updates ?
>
> we got few timeouts per day but not much, less then 10. I do not think
that timeouts will be root cause. I havent figured exact steps to reproduce
it (i havent even tried). We are reading at CL.ONE but cluster is well
synchronized and we are reading long time after writing - new value should
be present at all nodes allready.
>

Re: unknown exception with hector

[Deno Vichas]

i'm not sure if using framed transport is an option with hector.

what should i be in the logs looking for to find the cause of these 
dropped reads?


thanks,

On 5/24/2012 3:04 AM, aaron morton wrote:
Dropped read messages occur when the node could not process a read 
task within rpc_timeout. It generally means the cluster has been 
overwhelmed at some point:  too many requests, to much GC, compaction 
hurting, etc.


Check the server side logs for errors but I doubt it is related to the 
call stack below.


Have you confirmed that you are using framed transport on the client?

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 24/05/2012, at 5:52 PM, Deno Vichas wrote:

i've notice the my nodes seem to have a large (?, not really sure 
what acceptable numbers are) read dropped count from tpstats.  could 
they be related?


On 5/23/2012 2:55 AM, aaron morton wrote:

No sure but

   at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)


Looks like the client is not using framed transport. The server 
defaults to framed.


Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com 

On 23/05/2012, at 5:35 AM, Deno Vichas wrote:

could somebody clue me in to the cause of this exception?  i see 
these randomly.


AnalyzerService-2 2012-05-22 13:28:00,385 :: WARN 
 cassandra.connection.HConnectionManager  - Exception:
me.prettyprint.hector.api.exceptions.HectorTransportException: 
org.apache.thrift.transport.TTransportException
   at 
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:39)
   at 
me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:851)
   at 
me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:840)
   at 
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:99)
   at 
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:243)
   at 
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)
   at 
me.prettyprint.cassandra.service.KeyspaceServiceImpl.getColumn(KeyspaceServiceImpl.java:857)
   at 
me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:57)
   at 
me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(ThriftColumnQuery.java:52)
   at 
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
   at 
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
   at 
me.prettyprint.cassandra.model.thrift.ThriftColumnQuery.execute(ThriftColumnQuery.java:51)
   at 
com.stocktouch.dao.StockDaoImpl.getHistorical(StockDaoImpl.java:365)
   at 
com.stocktouch.dao.StockDaoImpl.getHistoricalQuote(StockDaoImpl.java:433)
   at 
com.stocktouch.service.StockHistoryServiceImpl.getHistoricalQuote(StockHistoryServiceImpl.java:480)
   at 
com.stocktouch.service.AnalyzerServiceImpl.getClose(AnalyzerServiceImpl.java:180)
   at 
com.stocktouch.service.AnalyzerServiceImpl.calcClosingPrices(AnalyzerServiceImpl.java:90)
   at 
com.stocktouch.service.AnalyzerServiceImpl.nightlyRollup(AnalyzerServiceImpl.java:66)
   at 
com.stocktouch.service.AnalyzerServiceImpl$2.run(AnalyzerServiceImpl.java:55)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

   at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.thrift.transport.TTransportException
   at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
   at 
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
   at 
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
   at 
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
   at 
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
   at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
   at 
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
   at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
   at 
org.apache.cassandra.thrift.Cassandra$Client.recv_get(Cassandra.java:509)
   at 
org.apache.cassandra.thrift.Cassandra$Client.get(Cassandra.java:492)
   at 
me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceServiceImpl.java:846)

   ... 20 more


thanks,
deno

about multitenant datamodel

[Toru Inoko]

Hi, all.

I'm designing data api service(like cassandra.io but not using dedicated  
server for each user) on cassandra 1.1 on which users can do DML/DDL  
method like cql.

Followings are api which users can use( almost same to cassandra api).
- create/read/delete ColumnFamilies/Rows/Columns

Now I'm thinking about multitenant datamodel on that.
My data model like the following.
I'm going to prepare a keyspace for each user as a user's tenant space.

| keyspace1 | --- | column family |
|(for user1)|  |
   ...

| keyspace2 | --- | column family |
|(for user2)|  |
   ...

Followings are my question!
- Is this data model a good for multitenant?
- Do a lot of keyspaces cause some problems? (If I have 1,000 users,  
cassandra creates 1,000 keyspaces...)


please, help.
thank you in advance.

Toru Inoko.

Help deleting data using cql v3

[Stephen Powis]

I have the following schema setup in cassandra 1.1 with cql 3:

CREATE TABLE testCol (
  my_id varchar,
  time_id TimeUUIDType,
  my_value int,
  PRIMARY KEY (my_id, time_id)
);

and the following data already inserted:

my_id| time_id   |
my_value |
1_71548 | 2fc39fa0-1dd5-11b2-9b6a-395f35722afe |  1|

but when I issue the following delete command using cqlsh -3, the data is
not removed.

delete from testCol where my_id='1_71548' and
time_id=2fc39fa0-1dd5-11b2-9b6a-395f35722afe;

Is it possible to remove data using the full composite key?  Or am I
formatting the UUID incorrectly?

Thanks!
Stephen

Re: Help deleting data using cql v3

[Roland Mechler]

This is a known issue, see
https://issues.apache.org/jira/browse/CASSANDRA-4193.

In the meantime, a workaround is to specify all the column names to be
deleted. I.e.,

  delete my_value from testCol where my_id='1_71548' and
time_id=2fc39fa0-1dd5-11b2-9b6a-395f35722afe;

should work.

(I had the same question answered here
http://www.datastax.com/dev/blog/whats-new-in-cql-3-0)

-Roland

On Thu, May 24, 2012 at 6:24 PM, Stephen Powis wrote:

> I have the following schema setup in cassandra 1.1 with cql 3:
>
> CREATE TABLE testCol (
>   my_id varchar,
>   time_id TimeUUIDType,
>   my_value int,
>   PRIMARY KEY (my_id, time_id)
> );
>
> and the following data already inserted:
>
> my_id| time_id   |
> my_value |
> 1_71548 | 2fc39fa0-1dd5-11b2-9b6a-395f35722afe |  1|
>
> but when I issue the following delete command using cqlsh -3, the data is
> not removed.
>
> delete from testCol where my_id='1_71548' and
> time_id=2fc39fa0-1dd5-11b2-9b6a-395f35722afe;
>
> Is it possible to remove data using the full composite key?  Or am I
> formatting the UUID incorrectly?
>
> Thanks!
> Stephen
>

Re: Help deleting data using cql v3

[Stephen Powis]

Thanks for the info Roland!  I guess I missed that bug report in my google
searching.

Stephen

On Fri, May 25, 2012 at 12:26 AM, Roland Mechler wrote:

> This is a known issue, see
> https://issues.apache.org/jira/browse/CASSANDRA-4193.
>
> In the meantime, a workaround is to specify all the column names to be
> deleted. I.e.,
>
>   delete my_value from testCol where my_id='1_71548' and
> time_id=2fc39fa0-1dd5-11b2-9b6a-395f35722afe;
>
> should work.
>
> (I had the same question answered here
> http://www.datastax.com/dev/blog/whats-new-in-cql-3-0)
>
> -Roland
>
> On Thu, May 24, 2012 at 6:24 PM, Stephen Powis 
> wrote:
>
>> I have the following schema setup in cassandra 1.1 with cql 3:
>>
>> CREATE TABLE testCol (
>>   my_id varchar,
>>   time_id TimeUUIDType,
>>   my_value int,
>>   PRIMARY KEY (my_id, time_id)
>> );
>>
>> and the following data already inserted:
>>
>> my_id| time_id   |
>> my_value |
>> 1_71548 | 2fc39fa0-1dd5-11b2-9b6a-395f35722afe |  1|
>>
>> but when I issue the following delete command using cqlsh -3, the data is
>> not removed.
>>
>> delete from testCol where my_id='1_71548' and
>> time_id=2fc39fa0-1dd5-11b2-9b6a-395f35722afe;
>>
>> Is it possible to remove data using the full composite key?  Or am I
>> formatting the UUID incorrectly?
>>
>> Thanks!
>> Stephen
>>
>
>

High CPU load on Cassandra Node

[Shubham Srivastava]

I have a multiDC ring with 6 nodes in each DC.

I have a single node which runs some jobs (including Hadoop Map-Reduce with 
PIG) every 15minutes.

Lately there has been high CPU load and memory issues on this node.

What I could see from Ganglia is high CPU load on this server and also number 
of TCP connection on port 9160 is around 600+ all the time.The distribution of 
these connections say that we have connections from this machine to other DC 
machines are around 90 odd each. For port 7000 its around 45.

Further I need to understand that for internal read/write does cassandra uses 
thrift for doing so over an rpc connection(port 9160) or 7000 as for inter node 
communication.May be that also could be a reason for so many connections on 
9160.

I have an 8Core machine with 14Gb RAM and 8Gb Heap.
rpc min and max threads are default and so are the other rpc based properties
RF:3 each DC  and Read/Write CL:1 and Read Repair Chance=0.1.
cassandra version is 0.8.6

Regards,
Shubham