Re: Continue running major compaction after switching to LeveledCompactionStrategy

[Lucas Fernandes Brunialti]

Hello,

I also have some doubts about changing to leveled compaction:

1) Is this change computationally expensive? My sstables have around 7gb of
data, I'm afraid the nodes won't handle the pressure of compactions, maybe
dying by OOM or getting an extremely high latency during the compactions...

2) How long does this transition takes? I mean, to finish the splitting of
these sstables and all the compactions needed... I wanted to know this to
make a fair comparison of which compaction algorithm is better for my data.

3) And finally, which would be an optimal size for the sstables, that LCS
parameter?

I'm running a 8 node cluster on aws (ec2 m1.xlarge), using ephemeral drives
and cassandra version 1.2.3.

I will really appreciate the help! :)

Lucas Brunialti.
Thanks much Rob!

Brian



--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Continue-running-major-compaction-after-switching-to-LeveledCompactionStrategy-tp7589839p7589846.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.

Re: row cache

[Faraaz Sareshwala]

After a bit of searching, I think I've found the answer I've been looking for. 
I guess I didn't search hard enough before sending out this email. Thank you 
all for the responses.

According to the datastax documentation [1], there are two types of row cache 
providers:

row_cache_provider
(Default: SerializingCacheProvider) Specifies what kind of implementation to 
use for the row cache.
SerializingCacheProvider: Serializes the contents of the row and stores it in 
native memory, that is, off the JVM Heap. Serialized rows take significantly 
less memory than live rows in the JVM, so you can cache more rows in a given 
memory footprint. Storing the cache off-heap means you can use smaller heap 
sizes, which reduces the impact of garbage collection pauses. It is valid to 
specify the fully-qualified class name to a class that 
implementsorg.apache.cassandra.cache.IRowCacheProvider.
ConcurrentLinkedHashCacheProvider: Rows are cached using the JVM heap, 
providing the same row cache behavior as Cassandra versions prior to 0.8.

The SerializingCacheProvider is 5 to 10 times more memory-efficient than 
ConcurrentLinkedHashCacheProvider for applications that are not blob-intensive. 
However, SerializingCacheProvider may perform worse in update-heavy workload 
situations because it invalidates cached rows on update instead of updating 
them in place as ConcurrentLinkedHashCacheProvider does.


The off-heap row cache provider does indeed invalidate rows. We're going to 
look into using the ConcurrentLinkedHashCacheProvider. Time to read some source 
code! :)

Faraaz

[1] 
http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__row_cache_provider




On Thursday, August 22, 2013 at 7:40 PM, Boris Yen wrote:

> If you are using off-heap memory for row cache, "all writes invalidate the 
> entire row" should be correct.
> 
> Boris
> 
> 
> On Fri, Aug 23, 2013 at 8:32 AM, Robert Coli  (mailto:rc...@eventbrite.com)> wrote:
> > On Wed, Aug 14, 2013 at 10:56 PM, Faraaz Sareshwala 
> > mailto:fsareshw...@quantcast.com)> wrote:
> > > All writes invalidate the entire row (updates thrown out the cached row)
> > This is not correct. Writes are added to the row, if it is in the row 
> > cache. If it's not in the row cache, the row is not added to the cache. 
> >  
> > Citation from jbellis on stackoverflow, because I don't have time to find a 
> > better one and the code is not obvious about it :
> > 
> > http://stackoverflow.com/a/12499422 
> > 
> > > I have yet to go through the source code for the row cache. I do plan to 
> > > do that. Can someone point me to documentation on the row cache 
> > > internals? All I've found online so far is small discussion about it and 
> > > how to enable it. 
> > 
> > There is no such documentation, or at least if it exists I am unaware of it.
> > 
> > In general, the rule of thumb is that the Row Cache should not be used 
> > unless the rows in question are : 
> > 
> > 1) Very hot in terms of access
> > 2) Uniform in size
> > 3) "Small"
> > 
> > =Rob  

Re: Decommission faster than bootstrap

[Boris Yen]

We are using 1.0. Our observation is that if you are using secondary index,
building secondary index after streaming is time consuming. And the
bootstrap needs to wait for the process of building secondary indexes to
complete.

I am not sure if this also applies to 1.1/1.2. You could set the log level
to debug to observe what is really going on when a node is bootstraping.

Boris


On Mon, Aug 19, 2013 at 6:19 AM, Rodrigo Felix <
rodrigofelixdealme...@gmail.com> wrote:

> Hi,
>
>I've noticed that, at least in my enviroment (Cassandra 1.1.12 running
> on Amazon EC2), decommission operations take about 3-4 minutes while
> bootstrap can take more than 20 minutes.
>What is the reason to have this time difference? For both operations,
> what it is time-consuming the data streaming from (or to) other node, right?
>Thanks in advance.
>
> Att.
>
> *Rodrigo Felix de Almeida*
> LSBD - Universidade Federal do Ceará
> Project Manager
> MBA, CSM, CSPO, SCJP
>

Re: row cache

[Boris Yen]

If you are using off-heap memory for row cache, "all writes invalidate the
entire row" should be correct.

Boris


On Fri, Aug 23, 2013 at 8:32 AM, Robert Coli  wrote:

> On Wed, Aug 14, 2013 at 10:56 PM, Faraaz Sareshwala <
> fsareshw...@quantcast.com> wrote:
>
>>
>>- All writes invalidate the entire row (updates thrown out the cached
>>row)
>>
>> This is not correct. Writes are added to the row, if it is in the row
> cache. If it's not in the row cache, the row is not added to the cache.
>
> Citation from jbellis on stackoverflow, because I don't have time to find
> a better one and the code is not obvious about it :
>
> http://stackoverflow.com/a/12499422
>
> I have yet to go through the source code for the row cache. I do plan to
>> do that. Can someone point me to documentation on the row cache internals?
>> All I've found online so far is small discussion about it and how to enable
>> it.
>>
>
> There is no such documentation, or at least if it exists I am unaware of
> it.
>
> In general, the rule of thumb is that the Row Cache should not be used
> unless the rows in question are :
>
> 1) Very hot in terms of access
> 2) Uniform in size
> 3) "Small"
>
> =Rob
>

Re: Continue running major compaction after switching to LeveledCompactionStrategy

[brianchang]

Thanks much Rob!

Brian



--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Continue-running-major-compaction-after-switching-to-LeveledCompactionStrategy-tp7589839p7589846.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: Continue running major compaction after switching to LeveledCompactionStrategy

[Robert Coli]

On Thu, Aug 22, 2013 at 5:24 PM, brianchang  wrote:

> One more follow-up question: can we freely switch back to
> SizeTieredCompactionStrategy later (and also resume running major
> compaction
> cron job), if we find LeveledCompactionStrategy does not end up with better
> performance (e.g., if we should experience those intensive I/O problems
> mentioned in the "when not to use Leveled Compaction" document)...?
>

Yes.

The switch back will likely trigger a bunch of compaction, but that's the
only side effect. If you're already hoping to do a major, that probably
isn't a problem for you. :D

=Rob

Re: row cache

[Robert Coli]

On Wed, Aug 14, 2013 at 10:56 PM, Faraaz Sareshwala <
fsareshw...@quantcast.com> wrote:

>
>- All writes invalidate the entire row (updates thrown out the cached
>row)
>
> This is not correct. Writes are added to the row, if it is in the row
cache. If it's not in the row cache, the row is not added to the cache.

Citation from jbellis on stackoverflow, because I don't have time to find a
better one and the code is not obvious about it :

http://stackoverflow.com/a/12499422

I have yet to go through the source code for the row cache. I do plan to do
> that. Can someone point me to documentation on the row cache internals? All
> I've found online so far is small discussion about it and how to enable it.
>

There is no such documentation, or at least if it exists I am unaware of it.

In general, the rule of thumb is that the Row Cache should not be used
unless the rows in question are :

1) Very hot in terms of access
2) Uniform in size
3) "Small"

=Rob

Re: Continue running major compaction after switching to LeveledCompactionStrategy

[brianchang]

Thanks much Edward!

One more follow-up question: can we freely switch back to
SizeTieredCompactionStrategy later (and also resume running major compaction
cron job), if we find LeveledCompactionStrategy does not end up with better
performance (e.g., if we should experience those intensive I/O problems
mentioned in the "when not to use Leveled Compaction" document)...?

Hopefully there will not be side effect by the rollback (e.g., the sstable
tree structure, etc)...

Brian



--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Continue-running-major-compaction-after-switching-to-LeveledCompactionStrategy-tp7589839p7589843.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: Continue running major compaction after switching to LeveledCompactionStrategy

[Edward Capriolo]

No . Leveled tables can not be manually compacted.

On Thursday, August 22, 2013, brianchang  wrote:
> Hi, Cassandra experts,
>
> Currently we are running major compaction (triggered daily by cron job),
as
> our application continue creating new columns in each row with old columns
> automatically expire/deleted by TTL.
>
> We are going to switch to using LeveledCompactionStrategy, and we are
> wondering if we need to continue the major compaction cron job? Or once
> being switched to using LeveledCompactionStrategy, we should not run major
> compaction anymore?
>
> Thanks,
> Brian
>
>
>
> --
> View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Continue-running-major-compaction-after-switching-to-LeveledCompactionStrategy-tp7589839.html
> Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
>

Re: write load while idle?

[Nate McCall]

Yep - OpsCenter stores it's own data in Cassandra. Thus the activity.

You could also turn on debug logging for StorageProxy on one of the nodes
if you really want to know.


On Thu, Aug 22, 2013 at 6:18 PM, Robert Coli  wrote:

> On Fri, Aug 16, 2013 at 4:46 PM, Keith Freeman <8fo...@gmail.com> wrote:
>
>> I have a 3-node cluster running 1.2.8, and with no clients connected (for
>> about an hour) opscenter is showing a heartbeat-like pattern for total
>> writes in the "Cluster Reads & Writes" panel on the dashboard ranging from
>> about 10/sec to 26/sec.  Total reads on the other hand are showing a
>> straight line hovering just barely above 0/sec.  There are no compactions,
>> etc. (the logs are totally quiet).
>>
>> Is this normal?  I'm new to cassandra so maybe this is normal but I
>> couldn't find an explanation in the docs and it seems pretty high for an
>> idle system.
>>
>
> That doesn't seem normal to me, but then I don't run Opscenter. Have you
> tried turning off OpsCenter to see if it changes?
>
> Also, those writes have to go.. somewhere? If you inspect memtables or
> flush, where do you see data being stored or flushed?
>
> =Rob
>
>

Re: Decommission faster than bootstrap

[Mike Heffner]

We've also noticed fairly poor streaming performance during a bootstrap
operation, albeit with 1.2.x. Streaming takes much longer than the physical
hardware capacity, even with the limits set high or off:
https://issues.apache.org/jira/browse/CASSANDRA-5726


On Sun, Aug 18, 2013 at 6:19 PM, Rodrigo Felix <
rodrigofelixdealme...@gmail.com> wrote:

> Hi,
>
>I've noticed that, at least in my enviroment (Cassandra 1.1.12 running
> on Amazon EC2), decommission operations take about 3-4 minutes while
> bootstrap can take more than 20 minutes.
>What is the reason to have this time difference? For both operations,
> what it is time-consuming the data streaming from (or to) other node, right?
>Thanks in advance.
>
> Att.
>
> *Rodrigo Felix de Almeida*
> LSBD - Universidade Federal do Ceará
> Project Manager
> MBA, CSM, CSPO, SCJP
>



-- 

  Mike Heffner 
  Librato, Inc.

Continue running major compaction after switching to LeveledCompactionStrategy

[brianchang]

Hi, Cassandra experts,

Currently we are running major compaction (triggered daily by cron job), as
our application continue creating new columns in each row with old columns
automatically expire/deleted by TTL.

We are going to switch to using LeveledCompactionStrategy, and we are
wondering if we need to continue the major compaction cron job? Or once
being switched to using LeveledCompactionStrategy, we should not run major
compaction anymore?

Thanks,
Brian



--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Continue-running-major-compaction-after-switching-to-LeveledCompactionStrategy-tp7589839.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Re: Random Distribution, yet Order Preserving Partitioner

[Takenori Sato]

Hi Nick,

> token and key are not same. it was like this long time ago (single MD5
assumed single key)

True. That reminds me of making a test with the latest 1.2 instead of our
current 1.0!

> if you want ordered, you probably can arrange your data in a way so you
can get it in ordered fashion.

Yeah, we have done for a long time. That's called a wide row, right? Or a
compound primary key.

It can handle some millions of columns, but not more like 10M. I mean, a
request for such a row concentrates on a particular node, so the
performance degrades.

> I also had idea for semi-ordered partitioner - instead of single MD5, to
have two MD5's.

Sounds interesting. But, we need a fully ordered result.

Anyway, I will try with the latest version.

Thanks,
Takenori


On Thu, Aug 22, 2013 at 6:12 PM, Nikolay Mihaylov  wrote:

> my five cents -
> token and key are not same. it was like this long time ago (single MD5
> assumed single key)
>
> if you want ordered, you probably can arrange your data in a way so you
> can get it in ordered fashion.
> for example long ago, i had single column family with single key and about
> 2-3 M columns - I do not suggest you to do it this way, because is wrong
> way, but it is easy to understand the idea.
>
> I also had idea for semi-ordered partitioner - instead of single MD5, to
> have two MD5's.
> then you can get semi-ordered ranges, e.g. you get ordered all cities in
> Canada, all cities in US and so on.
> however in this way things may get pretty non-ballanced
>
> Nick
>
>
>
>
>
> On Thu, Aug 22, 2013 at 11:19 AM, Takenori Sato wrote:
>
>> Hi,
>>
>> I am trying to implement a custom partitioner that evenly distributes,
>> yet preserves order.
>>
>> The partitioner returns a token by BigInteger as RandomPartitioner does,
>> while does a decorated key by string as OrderPreservingPartitioner does.
>> * for now, since IPartitioner does not support different types for
>> token and key, BigInteger is simply converted to string
>>
>> Then, I played around with cassandra-cli. As expected, in my 3 nodes test
>> cluster, get/set worked, but list(get_range_slices) didn't.
>>
>> This came from a challenge to overcome a wide row scalability. So, I want
>> to make it work!
>>
>> I am aware that some efforts are required to make get_range_slices work.
>> But are there any other critical problems? For example, it seems there is
>> an assumption that token and key are the same. If this is throughout the
>> whole C* code, this partitioner is not practical.
>>
>> Or have your tried something similar?
>>
>> I would appreciate your feedback!
>>
>> Thanks,
>> Takenori
>>
>
>

Re: write load while idle?

[Robert Coli]

On Fri, Aug 16, 2013 at 4:46 PM, Keith Freeman <8fo...@gmail.com> wrote:

> I have a 3-node cluster running 1.2.8, and with no clients connected (for
> about an hour) opscenter is showing a heartbeat-like pattern for total
> writes in the "Cluster Reads & Writes" panel on the dashboard ranging from
> about 10/sec to 26/sec.  Total reads on the other hand are showing a
> straight line hovering just barely above 0/sec.  There are no compactions,
> etc. (the logs are totally quiet).
>
> Is this normal?  I'm new to cassandra so maybe this is normal but I
> couldn't find an explanation in the docs and it seems pretty high for an
> idle system.
>

That doesn't seem normal to me, but then I don't run Opscenter. Have you
tried turning off OpsCenter to see if it changes?

Also, those writes have to go.. somewhere? If you inspect memtables or
flush, where do you see data being stored or flushed?

=Rob

Re: Decommission faster than bootstrap

[Robert Coli]

On Sun, Aug 18, 2013 at 3:19 PM, Rodrigo Felix <
rodrigofelixdealme...@gmail.com> wrote:

>I've noticed that, at least in my enviroment (Cassandra 1.1.12 running
> on Amazon EC2), decommission operations take about 3-4 minutes while
> bootstrap can take more than 20 minutes.
>What is the reason to have this time difference? For both operations,
> what it is time-consuming the data streaming from (or to) other node, right?
>

My first thought was that perhaps decommission ignores throttling of
streaming because the departing node doesn't need to be throttled. But the
target nodes for the streams do need to request stream throttling, so that
doesn't make sense to me. If you can reproduce this experience, I would
file a JIRA with the reproduction details at :

https://issues.apache.org/

=Rob

Re: aws VPC for cassandra

[Julio Quierati]

Hi,

create eni, set ip and attach in cassandra instance.
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html

[]'s


--
Julio Quierati
User Linux #492973

OpenPGP Key: 0xC9A064FA578E0D60
8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60


2013/8/22 Robert Coli 

> On Thu, Aug 22, 2013 at 11:13 AM, rash aroskar 
> wrote:
>
>> Has anyone used aws VPC for cassandra cluster? The static private ips of
>> VPC must be helpful in case of node replacement.
>> Please share any experiences related or suggest ideas for static ips in
>> ec2 for cassandra.
>>
>
> Who needs static ips? Changing the IP address of a Cassandra node is
> pretty easy.. ?
>
> (Seed lists do get stale, which kinda sucks.. but usually doesn't matter?)
>
> =Rob
>

Re: aws VPC for cassandra

[Robert Coli]

On Thu, Aug 22, 2013 at 11:13 AM, rash aroskar wrote:

> Has anyone used aws VPC for cassandra cluster? The static private ips of
> VPC must be helpful in case of node replacement.
> Please share any experiences related or suggest ideas for static ips in
> ec2 for cassandra.
>

Who needs static ips? Changing the IP address of a Cassandra node is pretty
easy.. ?

(Seed lists do get stale, which kinda sucks.. but usually doesn't matter?)

=Rob

aws VPC for cassandra

[rash aroskar]

Hello,
Has anyone used aws VPC for cassandra cluster? The static private ips of
VPC must be helpful in case of node replacement.
Please share any experiences related or suggest ideas for static ips in ec2
for cassandra.

-Rashmi

Re: Commitlog files not getting deleted

[Jay Svc]

its DSE 3.1 Cassandra 2.1



On Thu, Aug 22, 2013 at 10:28 AM, Robert Coli  wrote:

> On Thu, Aug 22, 2013 at 10:24 AM, Jay Svc  wrote:
>
>> In our cluster, the commit log is getting filled up as write progresses.
>> It is noticed that once the commit log is flushed to SSTable the commit log
>> files are not removed/deleted. The result of that the commit log volume is
>> getting filled with commit log files.
>>
>
> What version of Cassandra? There are some older versions with bugs like
> this, but haven't heard this symptom in a while...
>
> =Rob
>

Commitlog files not getting deleted

[Jay Svc]

Hi Users,

In our cluster, the commit log is getting filled up as write progresses. It
is noticed that once the commit log is flushed to SSTable the commit log
files are not removed/deleted. The result of that the commit log volume is
getting filled with commit log files.

Any reason why commit log is not cleared even after successful flush.

Thanks,
Jayant K Kenjale

Re: Commitlog files not getting deleted

[Robert Coli]

On Thu, Aug 22, 2013 at 10:24 AM, Jay Svc  wrote:

> In our cluster, the commit log is getting filled up as write progresses.
> It is noticed that once the commit log is flushed to SSTable the commit log
> files are not removed/deleted. The result of that the commit log volume is
> getting filled with commit log files.
>

What version of Cassandra? There are some older versions with bugs like
this, but haven't heard this symptom in a while...

=Rob

Re: node dead after restart

[Marcelo Elias Del Valle]

Ufff
I almost won the Charles Darwin prize now.
Thanks!
[]s


2013/8/22 Hiller, Dean 

> Isn't this the log file from 10.0.0.146??? And this 10.0.0.146 sees that
> 10.0.0.111 is up, then sees it dead and in the log we can see it bind with
> this line
>
> INFO 12:16:23,108 Binding thrift service to ip-10-0-0-146.ec2.internal/
> 10.0.0.146:9160
>
> What is the log file look like on 10.0.0.111?
>
> Thanks,
> Dean
>
> From: Marcelo Elias Del Valle  mvall...@gmail.com>>
> Reply-To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Date: Thursday, August 22, 2013 9:19 AM
> To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Subject: node dead after restart
>
> Hello,
>
> I am having a problem with a node in a test environment I have at
> amazon. I am using cassandra 1.2.3 in Amazon EC2. Here is my nodetool ring
> output:
>
> $ nodetool ring
> Note: Ownership information does not include topology; for complete
> information, specify a keyspace
>
> Datacenter: us-east
> ==
> Address RackStatus State   LoadOwns
>  Token
>
>  113427455640312821154458202479064646084
> 10.0.0.76   1b  Up Normal  31.34 MB33.33%
>  1808575600
> 10.0.0.1461b  Up Normal  34.24 MB33.33%
>56713727820156410577229101240436610842
> 10.0.0.111   1b  Down   Normal  21.19 MB33.33%
>  113427455640312821154458202479064646084
>
>  I logged in 10.0.0.111 machine and restarted cassandra, while looking
> at the log. Gossip protocol is still up, but the node starts and goes down
> just after it. Here is what I see in the logs:
>
> sudo tail /var/log/cassandra/output.log
>  INFO 12:16:23,084 Node /10.0.0.111 has restarted, now
> UP
>  INFO 12:16:23,095 InetAddress /10.0.0.111 is now UP
>  INFO 12:16:23,097 Node /10.0.0.111 state jump to
> normal
>  INFO 12:16:23,105 Not starting native transport as requested. Use JMX
> (StorageService->startNativeTransport()) to start it
>  INFO 12:16:23,108 Binding thrift service to ip-10-0-0-146.ec2.internal/
> 10.0.0.146:9160
>  INFO 12:16:23,137 Using TFramedTransport with a max frame size of
> 15728640 bytes.
>  INFO 12:16:23,143 Using synchronous/threadpool thrift server on
> ip-10-0-0-146.ec2.internal : 9160
>  INFO 12:16:23,143 Listening for thrift clients...
>  INFO 12:16:30,063 Saved local counter id:
> 76c1a930-a866-11e2-a3bd-831b111cd74c
>  INFO 12:16:32,860 InetAddress /10.0.0.111 is now dead.
>
>  I am having no clue of what is wrong. Any hint of what could I do to
> look for the problem?
>
> Best regards,
> --
> Marcelo Elias Del Valle
> http://mvalle.com - @mvallebr
>



-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Re: node dead after restart

[Hiller, Dean]

Isn't this the log file from 10.0.0.146??? And this 10.0.0.146 sees that 
10.0.0.111 is up, then sees it dead and in the log we can see it bind with this 
line

INFO 12:16:23,108 Binding thrift service to 
ip-10-0-0-146.ec2.internal/10.0.0.146:9160

What is the log file look like on 10.0.0.111?

Thanks,
Dean

From: Marcelo Elias Del Valle mailto:mvall...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Thursday, August 22, 2013 9:19 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: node dead after restart

Hello,

I am having a problem with a node in a test environment I have at amazon. I 
am using cassandra 1.2.3 in Amazon EC2. Here is my nodetool ring output:

$ nodetool ring
Note: Ownership information does not include topology; for complete 
information, specify a keyspace

Datacenter: us-east
==
Address RackStatus State   LoadOwns
Token
   
113427455640312821154458202479064646084
10.0.0.76   1b  Up Normal  31.34 MB33.33%  
1808575600
10.0.0.1461b  Up Normal  34.24 MB33.33%  
56713727820156410577229101240436610842
10.0.0.111   1b  Down   Normal  21.19 MB33.33%  
113427455640312821154458202479064646084

 I logged in 10.0.0.111 machine and restarted cassandra, while looking at 
the log. Gossip protocol is still up, but the node starts and goes down just 
after it. Here is what I see in the logs:

sudo tail /var/log/cassandra/output.log
 INFO 12:16:23,084 Node /10.0.0.111 has restarted, now UP
 INFO 12:16:23,095 InetAddress /10.0.0.111 is now UP
 INFO 12:16:23,097 Node /10.0.0.111 state jump to normal
 INFO 12:16:23,105 Not starting native transport as requested. Use JMX 
(StorageService->startNativeTransport()) to start it
 INFO 12:16:23,108 Binding thrift service to 
ip-10-0-0-146.ec2.internal/10.0.0.146:9160
 INFO 12:16:23,137 Using TFramedTransport with a max frame size of 15728640 
bytes.
 INFO 12:16:23,143 Using synchronous/threadpool thrift server on 
ip-10-0-0-146.ec2.internal : 9160
 INFO 12:16:23,143 Listening for thrift clients...
 INFO 12:16:30,063 Saved local counter id: 76c1a930-a866-11e2-a3bd-831b111cd74c
 INFO 12:16:32,860 InetAddress /10.0.0.111 is now dead.

 I am having no clue of what is wrong. Any hint of what could I do to look 
for the problem?

Best regards,
--
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

node dead after restart

[Marcelo Elias Del Valle]

Hello,

I am having a problem with a node in a test environment I have at
amazon. I am using cassandra 1.2.3 in Amazon EC2. Here is my nodetool ring
output:

$ nodetool ring
Note: Ownership information does not include topology; for complete
information, specify a keyspace

Datacenter: us-east
==
Address RackStatus State   LoadOwns
   Token

   113427455640312821154458202479064646084
10.0.0.76   1b  Up Normal  31.34 MB33.33%
 1808575600
10.0.0.1461b  Up Normal  34.24 MB33.33%
 56713727820156410577229101240436610842
10.0.0.111   1b  Down   Normal  21.19 MB33.33%
 113427455640312821154458202479064646084

 I logged in 10.0.0.111 machine and restarted cassandra, while looking
at the log. Gossip protocol is still up, but the node starts and goes down
just after it. Here is what I see in the logs:

sudo tail /var/log/cassandra/output.log
 INFO 12:16:23,084 Node /10.0.0.111 has restarted, now UP
 INFO 12:16:23,095 InetAddress /10.0.0.111 is now UP
 INFO 12:16:23,097 Node /10.0.0.111 state jump to normal
 INFO 12:16:23,105 Not starting native transport as requested. Use JMX
(StorageService->startNativeTransport()) to start it
 INFO 12:16:23,108 Binding thrift service to ip-10-0-0-146.ec2.internal/
10.0.0.146:9160
 INFO 12:16:23,137 Using TFramedTransport with a max frame size of 15728640
bytes.
 INFO 12:16:23,143 Using synchronous/threadpool thrift server on
ip-10-0-0-146.ec2.internal : 9160
 INFO 12:16:23,143 Listening for thrift clients...
 INFO 12:16:30,063 Saved local counter id:
76c1a930-a866-11e2-a3bd-831b111cd74c
 INFO 12:16:32,860 InetAddress /10.0.0.111 is now dead.

 I am having no clue of what is wrong. Any hint of what could I do to
look for the problem?

Best regards,
-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Re: how-to scan a table using CQL3

[Alain RODRIGUEZ]

Oops, I made a mistake thought I was paging on partition key when I
actually was paging on columns. No need of token and columns are ordered.
Sorry about bothering the ones who read this, it was a PEBCAK.

Alain




2013/8/21 Alain RODRIGUEZ 

> Hi, I am sorry about digging this up but I was in search of this kind of
> information and read this thread.
>
> How to make sure that the first rowkey you select has the smaller token ?
> I mean when you perform "select rowkey from my_table limit N;" can you have
> any data with any token or is data token ordered by default ?
>
> I tried it in dev and I have data with bigger token, but also with smaller
> token so I would have to do :
>
>  select rowkey from my_table where token(rowkey)
> > token(last_rowkey_returned) limit N;
>
> But also :
>
>  select rowkey from my_table where token(rowkey) <
> token(last_rowkey_returned) limit N;
>
> How to make sure you scan all your data, and only once, with CQL3 ? Am I
> misunderstanding or missing something ?
>
> Alain
>
> 2013/5/14 aaron morton 
>
>> select rowkey from my_table limit N;
>> while some_row_is_returned do
>>  select rowkey from my_table where token(rowkey) >
>> token(last_rowkey_returned) limit N;
>>
>> That should work for you.
>>
>> See
>> http://www.datastax.com/docs/1.2/cql_cli/using/paging#non-ordered-partitioner-paging
>>
>> Cheers
>>
>>-
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 11/05/2013, at 9:23 AM, Thorsten von Eicken 
>> wrote:
>>
>> Thanks, this is interesting, but if I'm not mistaken, Astyanax uses CQL2.
>> I'm trying to find a CQL3 solution on top the binary protocol. There has to
>> be a way to do this in CQL3...?
>> Thorsten
>>
>>
>>
>> On 5/10/2013 1:33 PM, Keith Wright wrote:
>>
>> What you are proposing should work and I started to implement that using
>> multiple threads over the token ranges but decided instead to use to
>> Astyanax's read all rows recipe as it does much of that already.  It
>> required some work to convert the composite CQL2 format returned from
>> Astayanx into what is expected for CQL3 but did work.  Here's an outline
>> of what you would do:
>>
>>
>>
>>
>>
>

how can i get the column value? Need help!.. cassandra 1.28 and pig 0.11.1

[Miguel Angel Martin junquera]

hi all:




I,m testing the new CqlStorage() with cassandra 1.28 and pig 0.11.1


I am using this sample data test:


http://frommyworkshop.blogspot.com.es/2013/07/hadoop-map-reduce-with-cassandra.html

And I load and dump data Righ with this script:

*rows = LOAD
'cql://keyspace1/test?page_size=1&split_size=4&where_clause=age%3D30' USING
CqlStorage();*
*
*
*dump rows;*
*describe rows;*
*
*

*resutls:

((id,6),(age,30),(title,QA))

((id,5),(age,30),(title,QA))

rows: {id: chararray,age: int,title: chararray}


*


But i can not  get  the column values

I try to define   another schemas in Load like I used with
cassandraStorage()

http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Cassandra-and-Pig-how-to-get-column-values-td5641158.html


example:

*rows = LOAD
'cql://keyspace1/test?page_size=1&split_size=4&where_clause=age%3D30' USING
CqlStorage() AS (columns: bag {T: tuple(name, value)});*


and I get this error:

*2013-08-22 12:24:45,426 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1031: Incompatable schema: left is
"columns:bag{T:tuple(name:bytearray,value:bytearray)}", right is
"id:chararray,age:int,title:chararray"*




I try to use, FLATTEN, SUBSTRING, SPLIT UDF`s but i have not get good
result:

Example:


   - when I flatten , I get a set of tuples like

*(title,QA)*

*(title,QA)*

*2013-08-22 12:42:20,673 [main] INFO
 org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1*

*A: {title: chararray}*



but i can get value QA

Sustring only works with title



example:

*B = FOREACH A GENERATE SUBSTRING(title,2,5);*
*
*
*dump B;*
*describe B;*
*
*
*
*

*results:*
*
*

*(tle)*
*(tle)*
*B: {chararray}*




i try, this like ERIC LEE inthe other mail  and have the same results:


 Anyways, what I really what is the column value, not the name. Is there a
way to do that? I listed all of the failed attempts I made below.

   - colnames = FOREACH cols GENERATE $1 and was told $1 was out of bounds.
   - casted = FOREACH cols GENERATE (tuple(chararray, chararray))$0; but
   all I got back were empty tuples
   - values = FOREACH cols GENERATE $0.$1; but I got an error telling me
   data byte array can't be casted to tuple


Please, I will appreciate any help


Regards

Re: Random Distribution, yet Order Preserving Partitioner

[Nikolay Mihaylov]

my five cents -
token and key are not same. it was like this long time ago (single MD5
assumed single key)

if you want ordered, you probably can arrange your data in a way so you can
get it in ordered fashion.
for example long ago, i had single column family with single key and about
2-3 M columns - I do not suggest you to do it this way, because is wrong
way, but it is easy to understand the idea.

I also had idea for semi-ordered partitioner - instead of single MD5, to
have two MD5's.
then you can get semi-ordered ranges, e.g. you get ordered all cities in
Canada, all cities in US and so on.
however in this way things may get pretty non-ballanced

Nick





On Thu, Aug 22, 2013 at 11:19 AM, Takenori Sato  wrote:

> Hi,
>
> I am trying to implement a custom partitioner that evenly distributes, yet
> preserves order.
>
> The partitioner returns a token by BigInteger as RandomPartitioner does,
> while does a decorated key by string as OrderPreservingPartitioner does.
> * for now, since IPartitioner does not support different types for
> token and key, BigInteger is simply converted to string
>
> Then, I played around with cassandra-cli. As expected, in my 3 nodes test
> cluster, get/set worked, but list(get_range_slices) didn't.
>
> This came from a challenge to overcome a wide row scalability. So, I want
> to make it work!
>
> I am aware that some efforts are required to make get_range_slices work.
> But are there any other critical problems? For example, it seems there is
> an assumption that token and key are the same. If this is throughout the
> whole C* code, this partitioner is not practical.
>
> Or have your tried something similar?
>
> I would appreciate your feedback!
>
> Thanks,
> Takenori
>

Random Distribution, yet Order Preserving Partitioner

[Takenori Sato]

Hi,

I am trying to implement a custom partitioner that evenly distributes, yet
preserves order.

The partitioner returns a token by BigInteger as RandomPartitioner does,
while does a decorated key by string as OrderPreservingPartitioner does.
* for now, since IPartitioner does not support different types for token
and key, BigInteger is simply converted to string

Then, I played around with cassandra-cli. As expected, in my 3 nodes test
cluster, get/set worked, but list(get_range_slices) didn't.

This came from a challenge to overcome a wide row scalability. So, I want
to make it work!

I am aware that some efforts are required to make get_range_slices work.
But are there any other critical problems? For example, it seems there is
an assumption that token and key are the same. If this is throughout the
whole C* code, this partitioner is not practical.

Or have your tried something similar?

I would appreciate your feedback!

Thanks,
Takenori