Re: Cassandra HANGS after some writes

[Naresh Yadav]

I made one single change in default cassandra.yaml, just to experiment.

native_transport_min_threads: *1*
native_transport_max_threads: *1*

with max one single thread for native protocol requests i noticed some
improvement, earlier with default yaml most of time it was failing after *
10K* combinations BUT with this it worked storing *30K* combinations out of
1lakh..

Please guide me further on this hint...

Naresh

On Tue, Aug 13, 2013 at 11:06 PM, Robert Coli  wrote:

> On Tue, Aug 13, 2013 at 10:34 AM, Andrew Cobley 
> wrote:
>
>> Has anyone ever done any performance comparisons of linux vs a headless
>> windows server ?
>>
>
> No, but given the number of linux specific optimizations in Cassandra, I
> would expect this to be no contest.
>
> =Rob
>

Re: Recommendation for hosting multi tenant clusters

[Ben Bromhead]

 http://www.mail-archive.com/user@cassandra.apache.org/msg11022.html sums it up 
pretty well. Optimised images and provisioned IOPS may help, but whatever way 
you spin it your reads and writes are still going out on the network somewhere.

EBS is like a giant SAN which will drop out at any second, take almost 
everything in your region down with it whilst simultaneously opening up a gate 
to hell that lets all sorts of unimaginable horrors into the world. 

Ok maybe not that bad, but network issues between ebs and your instances is 
painful. Whereas network issues with a single AZ can be dealt with in the 
course of normal cluster operations.

On a slight tangent, have a read of 
http://thelastpickle.com/2011/06/13/Down-For-Me/ which does an awesome job of 
explaining what will happen to your quorum reads and writes when a AWS AZ goes 
down (and you use ephemeral storage).

Cheers

Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359


On 14/08/2013, at 10:42 AM, Jon Haddad  wrote:

> I strongly recommend against EBS, even with optimized & ebs provisioned.  The 
> throughput you'll get from local drives is significantly better than what 
> you'll get with EBS (even 4K iops provisioned)
> 
> On Aug 13, 2013, at 2:10 PM, Rahul Gupta  wrote:
> 
>> I am working on requirement to host multi tenant Cassandra cluster (or set 
>> of clusters) on Amazon EC2 (AWS).
>>  
>> With everything else sorted out, I have below question where I am looking 
>> for recommendations:
>>  
>> Does Amazon’s recent support of EBS optimized images changes whole 
>> discussion around EBS vs. ephemeral drives and image size?
>>  
>> · Option 1: reserved m1.xlarge (4x420GB drives) is $0.187/hr
>> · Option 2: reserved m1.large EBS-optimized  is $0.119/hr 
>> (~$50/month less than m1.xlarge, but $168/month for 4x420 standard EBS 
>> volumes): costs $120/month more, but additional recovery options
>>  
>> Given Cassandra is designed to survive failures, combining replication 
>> factor 3 and backing-up to S3, I think should be enough for back up.
>>  
>> Please advise.
>>  
>> Thanks,
>> Rahul Gupta
>> DEKA Research & Development
>> 340 Commercial St  Manchester, NH  03101
>> P: 603.666.3908 extn. 6504 | C: 603.718.9676
>>  
>> This e-mail and the information, including any attachments, it contains are 
>> intended to be a confidential communication only to the person or entity to 
>> whom it is addressed and may contain information that is privileged. If the 
>> reader of this message is not the intended recipient, you are hereby 
>> notified that any dissemination, distribution or copying of this 
>> communication is strictly prohibited. If you have received this 
>> communication in error, please immediately notify the sender and destroy the 
>> original message.
>> 
>>  
>> 
>> This e-mail and the information, including any attachments, it contains are 
>> intended to be a confidential communication only to the person or entity to 
>> whom it is addressed and may contain information that is privileged. If the 
>> reader of this message is not the intended recipient, you are hereby 
>> notified that any dissemination, distribution or copying of this 
>> communication is strictly prohibited. If you have received this 
>> communication in error, please immediately notify the sender and destroy the 
>> original message.
>> 
>> Thank you.
>> 
>> Please consider the environment before printing this email.
> 

Re: Recommendation for hosting multi tenant clusters

[Jon Haddad]

I strongly recommend against EBS, even with optimized & ebs provisioned.  The 
throughput you'll get from local drives is significantly better than what 
you'll get with EBS (even 4K iops provisioned)

On Aug 13, 2013, at 2:10 PM, Rahul Gupta  wrote:

> I am working on requirement to host multi tenant Cassandra cluster (or set of 
> clusters) on Amazon EC2 (AWS).
>  
> With everything else sorted out, I have below question where I am looking for 
> recommendations:
>  
> Does Amazon’s recent support of EBS optimized images changes whole discussion 
> around EBS vs. ephemeral drives and image size?
>  
> · Option 1: reserved m1.xlarge (4x420GB drives) is $0.187/hr
> · Option 2: reserved m1.large EBS-optimized  is $0.119/hr (~$50/month 
> less than m1.xlarge, but $168/month for 4x420 standard EBS volumes): costs 
> $120/month more, but additional recovery options
>  
> Given Cassandra is designed to survive failures, combining replication factor 
> 3 and backing-up to S3, I think should be enough for back up.
>  
> Please advise.
>  
> Thanks,
> Rahul Gupta
> DEKA Research & Development
> 340 Commercial St  Manchester, NH  03101
> P: 603.666.3908 extn. 6504 | C: 603.718.9676
>  
> This e-mail and the information, including any attachments, it contains are 
> intended to be a confidential communication only to the person or entity to 
> whom it is addressed and may contain information that is privileged. If the 
> reader of this message is not the intended recipient, you are hereby notified 
> that any dissemination, distribution or copying of this communication is 
> strictly prohibited. If you have received this communication in error, please 
> immediately notify the sender and destroy the original message.
> 
>  
> 
> This e-mail and the information, including any attachments, it contains are 
> intended to be a confidential communication only to the person or entity to 
> whom it is addressed and may contain information that is privileged. If the 
> reader of this message is not the intended recipient, you are hereby notified 
> that any dissemination, distribution or copying of this communication is 
> strictly prohibited. If you have received this communication in error, please 
> immediately notify the sender and destroy the original message.
> 
> Thank you.
> 
> Please consider the environment before printing this email.

Re: Upgrade path to 2.0

[Robert Coli]

On Tue, Aug 13, 2013 at 2:58 PM, Randy Fradin wrote:

> If I'm reading this right, it seems CASSANDRA-5695 only bumps the
> upgrade path to 1.2.7. And is CASSANDRA-5845 still a concern if we
> don't make any schema changes while the cluster is in a split version
> state?


That is my understanding re: 5695, though 1.2.7 doesn't actually exist due
to being withdrawn as a result of a regression. Anything marked 1.2.7 is
effectively 1.2.8.

Per iamaleksey (#cassandra@freenode), the upgrade process to 2.0 creates
new schema, which is capable of being pulled by any 1.2 node in the same
cluster. So the two previously mentioned bugs preclude online ("rolling
restart") upgrade without first upgrading to 1.2.x >=1.2.9.

There are also third and fourth (related) issues, which aleksey just
confirmed are the *real* reason you *must* pass through 1.2.x >= 1.2.9, as
they preclude offline upgrade with the entire cluster down. You *must*
start your node on >= 1.2.9 < = 2.0.x, because the fixes occur at startup
time.

https://issues.apache.org/jira/browse/CASSANDRA-5800
https://issues.apache.org/jira/browse/CASSANDRA-5125

=Rob

Re: Upgrade path to 2.0

[Randy Fradin]

Thanks, I won't be upgrading to 2.0 any time soon but I would prefer
not to have to do a minor 1.2.8->1.2.9 upgrade before I do down the
road.

If I'm reading this right, it seems CASSANDRA-5695 only bumps the
upgrade path to 1.2.7. And is CASSANDRA-5845 still a concern if we
don't make any schema changes while the cluster is in a split version
state?

On Tue, Aug 13, 2013 at 2:41 PM, Robert Coli  wrote:
> On Tue, Aug 13, 2013 at 11:25 AM, Randy Fradin 
> wrote:
>>
>> According to the upgrade notes from NEWS.txt, under 2.0.0 it says:
>>
>> - Upgrading is ONLY supported from Cassandra 1.2.9 or later. This
>>   goes for sstable compatibility as well as network.  When
>>   upgrading from an earlier release, upgrade to 1.2.9 first and
>>   run upgradesstables before proceeding to 2.0.
>>
>> Can someone tell me what specifically makes 2.0.0 incompatible with
>> versions before 1.2.9? I have a 1.0.* cluster which I am looking to
>> upgrade to 1.2.*. Since it appears 1.2.9 is not even released yet
>> should I be waiting to upgrade?
>
>
> This is not the first time that NEWS.txt tells you something that raises
> more questions than it answers. I continue to maintain that it would be
> useful to link to JIRA ticket(s) which are the reason for instructions in
> NEWS.txt.
>
> That said, per driftx these are the two ~1.2.8 era tickets which mean you
> should upgrade from 1.2.9 or later.
>
> https://issues.apache.org/jira/browse/CASSANDRA-5696 : Upgrading to
> cassandra-1.2 with a dead LEFT state from 1.1 causes problems
> https://issues.apache.org/jira/browse/CASSANDRA-5845 : Don't pull schema
> from higher major nodes; don't push schema to lower major nodes
>
> Disclaimer : You probably don't want to upgrade a real cluster to 2.0
> anytime in the next 5 or so months.
>
> =Rob
>

Recommendation for hosting multi tenant clusters

[Rahul Gupta]

I am working on requirement to host multi tenant Cassandra cluster (or set of 
clusters) on Amazon EC2 (AWS).

With everything else sorted out, I have below question where I am looking for 
recommendations:

Does Amazon's recent support of EBS optimized images changes whole discussion 
around EBS vs. ephemeral drives and image size?



* Option 1: reserved m1.xlarge (4x420GB drives) is $0.187/hr

* Option 2: reserved m1.large EBS-optimized  is $0.119/hr (~$50/month 
less than m1.xlarge, but $168/month for 4x420 standard EBS volumes): costs 
$120/month more, but additional recovery options

Given Cassandra is designed to survive failures, combining replication factor 3 
and backing-up to S3, I think should be enough for back up.

Please advise.

Thanks,
Rahul Gupta
DEKA Research & Development
340 Commercial St  Manchester, NH  03101
P: 603.666.3908 extn. 6504 | C: 603.718.9676

This e-mail and the information, including any attachments, it contains are 
intended to be a confidential communication only to the person or entity to 
whom it is addressed and may contain information that is privileged. If the 
reader of this message is not the intended recipient, you are hereby notified 
that any dissemination, distribution or copying of this communication is 
strictly prohibited. If you have received this communication in error, please 
immediately notify the sender and destroy the original message.



This e-mail and the information, including any attachments, it contains are 
intended to be a confidential communication only to the person or entity to 
whom it is addressed and may contain information that is privileged. If the 
reader of this message is not the intended recipient, you are hereby notified 
that any dissemination, distribution or copying of this communication is 
strictly prohibited. If you have received this communication in error, please 
immediately notify the sender and destroy the original message.

Thank you.

Please consider the environment before printing this email.

token(), limit and wide rows

[Jan Algermissen]

HI,

ok, so I found token() [1], and that it is an option for paging through 
randomly partitioned data. 

I take it that combining token() and LIMIT is the CQL3 idiom for paging (set 
aside the fact that one shouldn't raelly want to page and use C*)

Now, when I page through a CF with wide rows, limitting each 'page' to, for 
example, 100 I end up in situations where not all 'sub'rows that have the same 
result for token() are returned because LIMIT chops off the result after 100 
'sub'rows, not neccessarily at the boundary to the next wide row.

Obvious ... but inconvenient.

The solution would be to throw away the last token returned (because it's wide 
row could have been chopped off) and do the next query with the token before.

So instead of doing

 SELECT * FROM users WHERE token(name) > token(last-name-of-prev-result) 
LIMIT 100;

I'd be doing

SELECT * FROM users WHERE token(name) > 
token(one-befoe-the-last-name-of-prev-result) LIMIT 100;


Question: Is that what I have to do or is there a way to make token() and limit 
work together to return complete wide rows?


Jan



[1] token() and how it relates to paging is actually quite hard to grasp from 
the docs.

Re: Upgrade path to 2.0

[Robert Coli]

On Tue, Aug 13, 2013 at 11:25 AM, Randy Fradin wrote:

> According to the upgrade notes from NEWS.txt, under 2.0.0 it says:
>
> - Upgrading is ONLY supported from Cassandra 1.2.9 or later. This
>   goes for sstable compatibility as well as network.  When
>   upgrading from an earlier release, upgrade to 1.2.9 first and
>   run upgradesstables before proceeding to 2.0.
>
> Can someone tell me what specifically makes 2.0.0 incompatible with
> versions before 1.2.9? I have a 1.0.* cluster which I am looking to
> upgrade to 1.2.*. Since it appears 1.2.9 is not even released yet
> should I be waiting to upgrade?
>

This is not the first time that NEWS.txt tells you something that raises
more questions than it answers. I continue to maintain that it would be
useful to link to JIRA ticket(s) which are the reason for instructions in
NEWS.txt.

That said, per driftx these are the two ~1.2.8 era tickets which mean you
should upgrade from 1.2.9 or later.

https://issues.apache.org/jira/browse/CASSANDRA-5696 : Upgrading to
cassandra-1.2 with a dead LEFT state from 1.1 causes problems
https://issues.apache.org/jira/browse/CASSANDRA-5845 : Don't pull schema
from higher major nodes; don't push schema to lower major nodes

Disclaimer : You probably don't want to upgrade a real cluster to 2.0
anytime in the next 5 or so months.

=Rob

Re: 0.7 node not starting

[Ashley Martens]

1) Can't
2) Interesting
3) Interesting
4) Remove all the Hints and OpsCenter data files

I used 4 to get it working.



On 13 August 2013 11:23, Robert Coli  wrote:

> On Tue, Aug 13, 2013 at 11:14 AM, Ashley Martens 
> wrote:
>
> > INFO [main] 2013-08-13 18:14:13,759 DatabaseDescriptor.java (line 181)
> DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> > Caused by: java.io.IOError: java.io.IOException: Map failed
>
> 1) Upgrade your ancient and broken version of Cassandra ASAP.
> 2) Try disabling mmap and see if your node starts.
> 3) If not, try dumping sstables with sstable2json until you find a corrupt
> one?
>
> =Rob
>
>

Re: For Multi Datacenter Geo redundancy which snitch works better

[Robert Coli]

On Tue, Aug 13, 2013 at 11:08 AM, Ken Schell wrote:

>  Could you enlighten me with issues we could expect if we went with
> the RackInferringSnitch? It seems the smiplest  to maintain when adding
> nodes.
>

Almost no one uses it, so you are not assured that it has been shown to
work as expected, at scale. The code, however, is pretty simple, so it
probably does.

As to issues... many people in many different environments do not have
control over their IP assignments or unlimited blocks of IPs to preallocate
for future expansion. This makes it potentially awkward to add new hosts in
the future. As you are @rackspace.com, perhaps you do have this control. In
that case, the RackInferringSnitch is probably fine.

That said, anyone using a rack aware snitch should read :

https://issues.apache.org/jira/browse/CASSANDRA-3810

=Rob

Upgrade path to 2.0

[Randy Fradin]

According to the upgrade notes from NEWS.txt, under 2.0.0 it says:

- Upgrading is ONLY supported from Cassandra 1.2.9 or later. This
  goes for sstable compatibility as well as network.  When
  upgrading from an earlier release, upgrade to 1.2.9 first and
  run upgradesstables before proceeding to 2.0.

Can someone tell me what specifically makes 2.0.0 incompatible with
versions before 1.2.9? I have a 1.0.* cluster which I am looking to
upgrade to 1.2.*. Since it appears 1.2.9 is not even released yet
should I be waiting to upgrade?

Re: 0.7 node not starting

[Robert Coli]

On Tue, Aug 13, 2013 at 11:14 AM, Ashley Martens wrote:

> INFO [main] 2013-08-13 18:14:13,759 DatabaseDescriptor.java (line 181)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> Caused by: java.io.IOError: java.io.IOException: Map failed

1) Upgrade your ancient and broken version of Cassandra ASAP.
2) Try disabling mmap and see if your node starts.
3) If not, try dumping sstables with sstable2json until you find a corrupt
one?

=Rob

0.7 node not starting

[Ashley Martens]

 INFO [main] 2013-08-13 18:14:13,674 AbstractCassandraDaemon.java (line
115) Heap size: 8568963072/8568963072
 INFO [main] 2013-08-13 18:14:13,674 AbstractCassandraDaemon.java (line
116) Classpath:
/usr/share/cassandra/antlr-3.1.3.jar:/usr/share/cassandra/apache-cassandra-0.7.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/avro-1.4.0-fixes.jar:/usr/share/cassandra/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/commons-cli-1.1.jar:/usr/share/cassandra/commons-codec-1.2.jar:/usr/share/cassandra/commons-collections-3.2.1.jar:/usr/share/cassandra/commons-lang-2.4.jar:/usr/share/cassandra/concurrentlinkedhashmap-lru-1.1.jar:/usr/share/cassandra/guava-r05.jar:/usr/share/cassandra/high-scale-lib.jar:/usr/share/cassandra/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/jetty-6.1.21.jar:/usr/share/cassandra/jetty-util-6.1.21.jar:/usr/share/cassandra/jline-0.9.94.jar:/usr/share/cassandra/json-simple-1.1.jar:/usr/share/cassandra/jug-2.0.0.jar:/usr/share/cassandra/libthrift-0.5.jar:/usr/share/cassandra/log4j-1.2.16.jar:/usr/share/cassandra/servlet-api-2.5-20081211.jar:/usr/share/cassandra/slf4j-api-1.6.1.jar:/usr/share/cassandra/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/snakeyaml-1.6.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar
 INFO [main] 2013-08-13 18:14:13,675 CLibrary.java (line 61) JNA not found.
Native methods will be disabled.
 INFO [main] 2013-08-13 18:14:13,682 DatabaseDescriptor.java (line 121)
Loading settings from file:/etc/cassandra/cassandra.yaml
 INFO [main] 2013-08-13 18:14:13,759 DatabaseDescriptor.java (line 181)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap




java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
Caused by: java.io.IOError: java.io.IOException: Map failed
at
org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:172)
at
org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:149)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:325)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:191)
at
org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:233)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:492)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:462)
at org.apache.cassandra.db.Table.initCf(Table.java:309)
at org.apache.cassandra.db.Table.(Table.java:246)
at org.apache.cassandra.db.Table.open(Table.java:106)
at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:205)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:148)
at
org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:237)
... 5 more
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:164)
... 17 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
... 18 more
Cannot load daemon
Service exit with a return value of 3

Re: For Multi Datacenter Geo redundancy which snitch works better

[Ken Schell]

Rob,

Could you enlighten me with issues we could expect if we went with the 
RackInferringSnitch? It seems the smiplest  to maintain when adding nodes.



From: Robert Coli mailto:rc...@eventbrite.com>>
Date: Monday, August 12, 2013 12:44 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Cc: ken Schell mailto:ken.sch...@rackspace.com>>
Subject: Re: For Multi Datacenter Geo redundancy which snitch works better

On Mon, Aug 12, 2013 at 7:06 AM, Ken Schell 
mailto:ken.sch...@rackspace.com>> wrote:
I'm building a two datacenter cluster for Geo redundancy, each with a minimum 
of 12 nodes.

Which Snitch would you recommend, PropertyFileSnitch or RackInferringSnitch?

I'd probably use GossipingPropertyFileSnitch before PropertyFileSnitch. I would 
almost certainly not use RackInferringSnitch.

https://issues.apache.org/jira/browse/CASSANDRA-1974

http://www.datastax.com/docs/1.1/cluster_architecture/replication

=Rob

Re: Cassandra HANGS after some writes

[Robert Coli]

On Tue, Aug 13, 2013 at 10:34 AM, Andrew Cobley wrote:

> Has anyone ever done any performance comparisons of linux vs a headless
> windows server ?
>

No, but given the number of linux specific optimizations in Cassandra, I
would expect this to be no contest.

=Rob

Re: What's difference between node tool removenode & decommission in 1.2.x

[Robert Coli]

On Tue, Aug 13, 2013 at 10:06 AM, Jasdeep Hundal  wrote:

> Removenode will stream the data that node is responsible for from other
> replicas, and AFAIK is generally used when a node is offline and cannot be
> brought back up.
>

removetoken should only be used if the departing node is unavailable or
might have corrupt data on disk. Removetoken involves a (small, in
practice) risk that the node being removetokened had the only copy of a
given piece of data.

=Rob

Re: Cassandra HANGS after some writes

[Andrew Cobley]

Agreed with Alain,

There are a number of reasons that he may be running on windows, a development 
environment comes to mind,or simply to learn about C*

BTW
Has anyone ever done any performance comparisons of linux vs a headless windows 
server ?

Andy

On 13 Aug 2013, at 17:07, Alain RODRIGUEZ 
mailto:arodr...@gmail.com>> wrote:

@Kanwar Sangha

"Cassandra on windows ? Please install Linux !"

Useful comment, please spare your time and stop that troll.

He surely have his reason to use windows (I suppose it is a dev constraint or 
choice). Anyway, C* is available in windows so it should work. Comments like 
"windows sucks, go linux or macOS", are not going to solve his issue. If 
Cassandra can't be run on windows, just don't package Cassandra for windows.

We just can recommend Naresh *not* to use Windows as the OS for your production 
nodes.

Alain


2013/8/13 Kanwar Sangha mailto:kan...@mavenir.com>>
Cassandra on windows ? Please install Linux !


From: Romain HARDOUIN 
[mailto:romain.hardo...@urssaf.fr]
Sent: 13 August 2013 10:17
To: user@cassandra.apache.org
Subject: Re: Cassandra HANGS after some writes

Naresh,

My two cents is that you should run Cassandra on a Linux VM.
Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to many 
people here.

Cheers

Alexis Rodríguez 
mailto:arodrig...@inconcertcc.com>> a écrit sur 
13/08/2013 16:50:42 :

> De : Alexis Rodríguez 
> mailto:arodrig...@inconcertcc.com>>
> A : user@cassandra.apache.org,
> Date : 13/08/2013 16:51
> Objet : Re: Cassandra HANGS after some writes
>
> Naresh,
>
> Windows is not my cup of tea. May be someone else has more
> experience using the Redmond's prodigy child.
>
> cheers, and good luck



The University of Dundee is a registered Scottish Charity, No: SC015096

Re: Cassandra HANGS after some writes

[Naresh Yadav]

Hi all,

I started cassandra few weeks back and i am on development enviornment, it
will take months for production as everything in development.But i will
spend time and setup one machine with UBuntu and will check if similar
problem comes or not...Also i had started hands on Hadoop then linux would
be must for me on production..

Till then if anybody can give me some pointers to try on windows parallely
as my most of team do not familiar
with linux enviornment thats why started on Windows.

Thanks
Naresh

On Tue, Aug 13, 2013 at 9:37 PM, Alain RODRIGUEZ  wrote:

> @Kanwar Sangha
>
> "Cassandra on windows ? Please install Linux !"
>
> Useful comment, please spare your time and stop that troll.
>
> He surely have his reason to use windows (I suppose it is a dev constraint
> or choice). Anyway, C* is available in windows so it should work. Comments
> like "windows sucks, go linux or macOS", are not going to solve his issue.
> If Cassandra can't be run on windows, just don't package Cassandra for
> windows.
>
> We just can recommend Naresh *not* to use Windows as the OS for your
> production nodes.
>
> Alain
>
>
> 2013/8/13 Kanwar Sangha 
>
>>  Cassandra on windows ? Please install Linux ! 
>>
>> ** **
>>
>> ** **
>>
>> *From:* Romain HARDOUIN [mailto:romain.hardo...@urssaf.fr]
>> *Sent:* 13 August 2013 10:17
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Cassandra HANGS after some writes
>>
>> ** **
>>
>> Naresh,
>>
>> My two cents is that you should run Cassandra on a Linux VM.
>> Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to
>> many people here.
>>
>> Cheers
>>
>> Alexis Rodríguez  a écrit sur 13/08/2013
>> 16:50:42 :
>>
>> > De : Alexis Rodríguez 
>> > A : user@cassandra.apache.org,
>> > Date : 13/08/2013 16:51
>> > Objet : Re: Cassandra HANGS after some writes
>> >
>> > Naresh,
>> >
>> > Windows is not my cup of tea. May be someone else has more
>> > experience using the Redmond's prodigy child.
>> >
>> > cheers, and good luck 
>>
>
>

Re: What's difference between node tool removenode & decommission in 1.2.x

[Frank Stutz]

Thanks for clarification.   ;-)


--Frank
On Aug 13, 2013, at 10:06 AM, Jasdeep Hundal 
mailto:dsjas...@gmail.com>> wrote:

Decommission moves the data from the node being decommissioned to the other 
nodes that will now have ownership over the data.

Removenode will stream the data that node is responsible for from other 
replicas, and AFAIK is generally used when a node is offline and cannot be 
brought back up.

I think you'll want to go with calling 'nodetool decommission' on the node 
you're trying to remove here.

Jasdeep


On Tue, Aug 13, 2013 at 10:01 AM, Frank Stutz 
mailto:frank.st...@xoom.com>> wrote:
I'm a little bit confused on why the nodetool utility has a decommission and a 
removenode command.  I speculate that decommission command just informs the 
other nodes that I'm going offline and does not move the data off that node. 
Whereas removenode resync's the data/tokens to the other nodes.   Although 
maybe I'm way off.

In short, I'm trying to remove a node and want to know the a good safe way to 
make sure its tokens are moved to other nodes.  I will note that the other 
nodes should already have the data, but I always prefer the safest way to 
insure data is not lost.

--Frank Stutz
-
The information transmitted in this email is intended only for the person or 
entity to which it is addressed, and may contain material confidential to Xoom 
Corporation, and/or its subsidiary, 
buyindiaonline.com Inc.  Any review, 
retransmission, dissemination or other use of, or taking of any action in 
reliance upon, this information by persons or entities other than the intended 
recipient(s) is prohibited.   If you received this email in error, please 
contact the sender and delete the material from your files.






-
The information transmitted in this email is intended only for the person or 
entity to which it is addressed, and may contain material confidential to Xoom 
Corporation, and/or its subsidiary, buyindiaonline.com Inc.  Any review, 
retransmission, dissemination or other use of, or taking of any action in 
reliance upon, this information by persons or entities other than the intended 
recipient(s) is prohibited.   If you received this email in error, please 
contact the sender and delete the material from your files.

Re: What's difference between node tool removenode & decommission in 1.2.x

[Jasdeep Hundal]

Decommission moves the data from the node being decommissioned to the other
nodes that will now have ownership over the data.

Removenode will stream the data that node is responsible for from other
replicas, and AFAIK is generally used when a node is offline and cannot be
brought back up.

I think you'll want to go with calling 'nodetool decommission' on the node
you're trying to remove here.

Jasdeep


On Tue, Aug 13, 2013 at 10:01 AM, Frank Stutz  wrote:

> I'm a little bit confused on why the nodetool utility has a decommission
> and a removenode command.  I speculate that decommission command just
> informs the other nodes that I'm going offline and does not move the data
> off that node. Whereas removenode resync's the data/tokens to the other
> nodes.   Although maybe I'm way off.
>
> In short, I'm trying to remove a node and want to know the a good safe way
> to make sure its tokens are moved to other nodes.  I will note that the
> other nodes should already have the data, but I always prefer the safest
> way to insure data is not lost.
>
> --Frank Stutz
>
> -
> The information transmitted in this email is intended only for the person
> or entity to which it is addressed, and may contain material confidential
> to Xoom Corporation, and/or its subsidiary, buyindiaonline.com Inc.  Any
> review, retransmission, dissemination or other use of, or taking of any
> action in reliance upon, this information by persons or entities other than
> the intended recipient(s) is prohibited.   If you received this email in
> error, please contact the sender and delete the material from your files.
>
>
>

What's difference between node tool removenode & decommission in 1.2.x

[Frank Stutz]

I'm a little bit confused on why the nodetool utility has a decommission and a 
removenode command.  I speculate that decommission command just informs the 
other nodes that I'm going offline and does not move the data off that node. 
Whereas removenode resync's the data/tokens to the other nodes.   Although 
maybe I'm way off.

In short, I'm trying to remove a node and want to know the a good safe way to 
make sure its tokens are moved to other nodes.  I will note that the other 
nodes should already have the data, but I always prefer the safest way to 
insure data is not lost.

--Frank Stutz
-
The information transmitted in this email is intended only for the person or 
entity to which it is addressed, and may contain material confidential to Xoom 
Corporation, and/or its subsidiary, buyindiaonline.com Inc.  Any review, 
retransmission, dissemination or other use of, or taking of any action in 
reliance upon, this information by persons or entities other than the intended 
recipient(s) is prohibited.   If you received this email in error, please 
contact the sender and delete the material from your files.

Re: HsHa

[Alain RODRIGUEZ]

I have this anytime I try to switch to "hsha" since 0.8.

Always kept "sync" for this reason. Thought I was alone with this bug since
I never had any clue about this on the mailing list.

So +1.

Alain


2013/8/13 Christopher Wirt 

> Hello,
>
> ** **
>
> I was trying out the hsha thrift server implementation and found that I
> get a fair amount of these appearing in the server logs.
>
> ** **
>
> ERROR [Selector-Thread-9] 2013-08-13 15:39:10,433 TNonblockingServer.java
> (line 468) Read an invalid frame size of 0. Are you using TFramedTransport
> on the client side?
>
> ERROR [Selector-Thread-9] 2013-08-13 15:39:11,499 TNonblockingServer.java
> (line 468) Read an invalid frame size of 0. Are you using TFramedTransport
> on the client side?
>
> ERROR [Selector-Thread-9] 2013-08-13 15:39:11,695 TNonblockingServer.java
> (line 468) Read an invalid frame size of 0. Are you using TFramedTransport
> on the client side?
>
> ERROR [Selector-Thread-9] 2013-08-13 15:39:12,562 TNonblockingServer.java
> (line 468) Read an invalid frame size of 0. Are you using TFramedTransport
> on the client side?
>
> ERROR [Selector-Thread-1] 2013-08-13 15:39:12,660 TNonblockingServer.java
> (line 468) Read an invalid frame size of 0. Are you using TFramedTransport
> on the client side?
>
> ERROR [Selector-Thread-9] 2013-08-13 15:39:13,496 TNonblockingServer.java
> (line 468) Read an invalid frame size of 0. Are you using TFramedTransport
> on the client side?
>
> ERROR [Selector-Thread-9] 2013-08-13 15:39:14,281 TNonblockingServer.java
> (line 468) Read an invalid frame size of 0. Are you using TFramedTransport
> on the client side?
>
> ** **
>
> Anyone seen this message before? know what it means? or issues it could
> hide?
>
> ** **
>
> https://issues.apache.org/jira/browse/CASSANDRA-4573 
>
> in the comments suggests it might be a 10 client timeout
>
> but looking at JMX client stats the max value for read/write/slice is well
> below 10secs
>
> ** **
>
> ** **
>
> I’m using 1.2.8 on centos
>
> ** **
>
> ** **
>
> Cheers,
>
> Chris
>

Re: Cassandra HANGS after some writes

[Alain RODRIGUEZ]

@Kanwar Sangha

"Cassandra on windows ? Please install Linux !"

Useful comment, please spare your time and stop that troll.

He surely have his reason to use windows (I suppose it is a dev constraint
or choice). Anyway, C* is available in windows so it should work. Comments
like "windows sucks, go linux or macOS", are not going to solve his issue.
If Cassandra can't be run on windows, just don't package Cassandra for
windows.

We just can recommend Naresh *not* to use Windows as the OS for your
production nodes.

Alain


2013/8/13 Kanwar Sangha 

>  Cassandra on windows ? Please install Linux ! 
>
> ** **
>
> ** **
>
> *From:* Romain HARDOUIN [mailto:romain.hardo...@urssaf.fr]
> *Sent:* 13 August 2013 10:17
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra HANGS after some writes
>
> ** **
>
> Naresh,
>
> My two cents is that you should run Cassandra on a Linux VM.
> Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to
> many people here.
>
> Cheers
>
> Alexis Rodríguez  a écrit sur 13/08/2013
> 16:50:42 :
>
> > De : Alexis Rodríguez 
> > A : user@cassandra.apache.org,
> > Date : 13/08/2013 16:51
> > Objet : Re: Cassandra HANGS after some writes
> >
> > Naresh,
> >
> > Windows is not my cup of tea. May be someone else has more
> > experience using the Redmond's prodigy child.
> >
> > cheers, and good luck 
>

HsHa

[Christopher Wirt]

Hello,

 

I was trying out the hsha thrift server implementation and found that I get
a fair amount of these appearing in the server logs.

 

ERROR [Selector-Thread-9] 2013-08-13 15:39:10,433 TNonblockingServer.java
(line 468) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

ERROR [Selector-Thread-9] 2013-08-13 15:39:11,499 TNonblockingServer.java
(line 468) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

ERROR [Selector-Thread-9] 2013-08-13 15:39:11,695 TNonblockingServer.java
(line 468) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

ERROR [Selector-Thread-9] 2013-08-13 15:39:12,562 TNonblockingServer.java
(line 468) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

ERROR [Selector-Thread-1] 2013-08-13 15:39:12,660 TNonblockingServer.java
(line 468) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

ERROR [Selector-Thread-9] 2013-08-13 15:39:13,496 TNonblockingServer.java
(line 468) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

ERROR [Selector-Thread-9] 2013-08-13 15:39:14,281 TNonblockingServer.java
(line 468) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

 

Anyone seen this message before? know what it means? or issues it could
hide?

 

https://issues.apache.org/jira/browse/CASSANDRA-4573 

in the comments suggests it might be a 10 client timeout

but looking at JMX client stats the max value for read/write/slice is well
below 10secs

 

 

I'm using 1.2.8 on centos

 

 

Cheers,

Chris

RE: Cassandra HANGS after some writes

[Kanwar Sangha]

Cassandra on windows ? Please install Linux !


From: Romain HARDOUIN [mailto:romain.hardo...@urssaf.fr]
Sent: 13 August 2013 10:17
To: user@cassandra.apache.org
Subject: Re: Cassandra HANGS after some writes

Naresh,

My two cents is that you should run Cassandra on a Linux VM.
Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to many 
people here.

Cheers

Alexis Rodríguez 
mailto:arodrig...@inconcertcc.com>> a écrit sur 
13/08/2013 16:50:42 :

> De : Alexis Rodríguez 
> mailto:arodrig...@inconcertcc.com>>
> A : user@cassandra.apache.org,
> Date : 13/08/2013 16:51
> Objet : Re: Cassandra HANGS after some writes
>
> Naresh,
>
> Windows is not my cup of tea. May be someone else has more
> experience using the Redmond's prodigy child.
>
> cheers, and good luck

Re: Cassandra HANGS after some writes

[Romain HARDOUIN]

Naresh, 

My two cents is that you should run Cassandra on a Linux VM.
Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to 
many people here.

Cheers

Alexis Rodríguez  a écrit sur 13/08/2013 
16:50:42 :

> De : Alexis Rodríguez 
> A : user@cassandra.apache.org, 
> Date : 13/08/2013 16:51
> Objet : Re: Cassandra HANGS after some writes
> 
> Naresh,
> 
> Windows is not my cup of tea. May be someone else has more 
> experience using the Redmond's prodigy child.
> 
> cheers, and good luck

Re: C* 1.0.6 to 1.1.12: upgradesstables or scrub?

[Romain HARDOUIN]

Yes, forgot to mention we use LCS in 1.0.6.
Thank you.

Jeremy Hanna  a écrit sur 13/08/2013 16:44:56 
:

> De : Jeremy Hanna 
> A : user@cassandra.apache.org, 
> Date : 13/08/2013 16:45
> Objet : Re: C* 1.0.6 to 1.1.12: upgradesstables or scrub?
> 
> If you were using leveled compaction on any column families in 1.0, 
> you'll need to run offline scrub on those column families.

Re: Cassandra HANGS after some writes

[Alexis Rodríguez]

Naresh,

Windows is not my cup of tea. May be someone else has more experience using
the Redmond's prodigy child.

cheers, and good luck


On Tue, Aug 13, 2013 at 11:21 AM, Naresh Yadav  wrote:

> Hi Alex,
>
> Yes i am testing in development environment of Windows 7 64bit.
> I left default yaml then cassandra created "var" folder and created data,
> log, cache folders in it.I tried commit log on different harddisk but
> this problem not solved with thatI guess this problem is somewhat
> related to deadlock in Native Transport threads...thats why cassandra is
> hanging indefinitly..
>
> Naresh
>
> On Tue, Aug 13, 2013 at 7:21 PM, Alexis Rodríguez <
> arodrig...@inconcertcc.com> wrote:
>
>> Naresh, are you deploying cassandra in windows?
>>
>> If that is the case you may need to change the data and commitlog
>> directories in cassandra.yaml. Also you should check the log directories.
>>
>> See the section 2.1  http://wiki.apache.org/cassandra/GettingStarted
>>
>>
>> On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav wrote:
>>
>>>
>>> Thanks Alain, will avoid capsi am newbie to cassandra, just started
>>> using 2 weeks back..
>>>
>>> Here are JConsole screenshots just 5mins after cassandra freezed :
>>>
>>> http://i.imgur.com/3oUBjKU.png
>>> http://i.imgur.com/2O4PrKb.png
>>> http://i.imgur.com/zxhFzr1.png   4:05 is time cassandra
>>> freezed thats why decline in no of threads
>>> http://i.imgur.com/ScgAciv.png
>>> Uploaded complete system.log of cassandra till freeze :
>>> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>>>
>>> Observation : As in my usecase i am storing 1lakh
>>> combinations(527insert,506update,954select) each parallel by 100 threads in
>>> batch of 1000...
>>> Sometimes it works till 1000 batch then hangs but sometimes it completes
>>> 1 then hangs and once even worked for more than lakh
>>> Same hardware Same settings of cassandra i see random behaviour of
>>> performance..
>>>
>>> Thanks
>>> Naresh
>>>
>>>
>>> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ wrote:
>>>
 Hi Naresh.

 First thing, there is no need of caps in here. People reading this ML
 is here to help when they have time and skills enough to do so. So please,
 chill out and do not use caps to show how much desperate you are.

 Concerning your problem, the only abnormal thing I was able to find in
 your logs is


1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
SSTableDeletingTask.java (line 72) Unable to delete

 \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
(it will be removed on server restart; we'll also retry after GC)


 I don't think this should keep C* hanging.

 Do you have something on kernel logs ?

 Do you have monitor any metrics like disk throughput / heap used / cpu
 load / iowait which are known as being bottlenecks / pertinent metrics ?

 Alain


 2013/8/13 Naresh Yadav 

>
> Hi All,
>
> I have single node cassandra using  CQL using datastax java driver
> 1.0.1 and cassandra verison 1.2.6.
>
> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
> processor.. DEFAULT cassandra.yaml no change done by me.
> -Xms1G^
>  -Xmx12G^ no other change in cassandra.bat
>
> *Problem :
> _
>  *cassandra Freezes after some writes and i see no action on
> cassandra console for an hour...all Native_Transport threads are also
> killedmy program keeps running NO ERROR comes...when i connect 
> with
> cql that works In start it creates 16 NativeTransport threads and
> after 10-15 minutes Total threads goes to 128...Just before it hangs, With
> JCONSOLE when i see Native_Transport threads then i find most of them in
> state as :
>
> http://pastebin.com/DeShpHtP
>
> *Load on cassandra : *
> ___
> i am running a usecase which stores Combinations(my project
> terminology) in cassandraCurrently testing storing 2.5 lakh
> combinations with 100 parallel threads..each thread storing one
> combination...real case i need to support of many CRORES but that would
> need different hardware and multi node cluster...
>
> In Storing ONE combination takes around 2sec and involves :
>
> 527 INSERT INTO queries 
>
> 506 UPDATE queries 
>
> 954 SELECT queries 
>
> 100 parallel threads parallely storing 100 combinations
> *MY CASSANDRA LOGS :*
>
> http://pastebin.com/CnNvA9x3
>
> Please look last 100-200 lines of log because that is time it freezed
>
>
> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>


>>>
>>>
>>>
>>
>
>
>

Re: C* 1.0.6 to 1.1.12: upgradesstables or scrub?

[Jeremy Hanna]

If you were using leveled compaction on any column families in 1.0, you'll need 
to run offline scrub on those column families.

On 13 Aug 2013, at 15:38, Romain HARDOUIN  wrote:

> Hi all, 
> 
> We are migrating from C* 1.0.6 to 1.1.12 and after reading DataStax 
> documentation (http://www.datastax.com/docs/1.1/install/upgrading) we are a 
> bit confused. 
> The section "Completing the Upgrade", step 6, points out: 
> 
> If you are upgrading from Cassandra 1.1.3 or earlier to Cassandra 1.1.5 
> or later, skip steps 7 and 8 of this procedure and go to Completing the 
> upgrade from Cassandra 1.1.3 or earlier to Cassandra 1.1.5 or later. 
> --> i.e. run an offline scrub 
> 
> However according to the step 8: 
> 
> After upgrading, run nodetool upgradesstables against each node before 
> running repair, moving nodes, or adding new ones. 
> If you are using Cassandra 1.0.3 and earlier, use nodetool scrub instead 
> of nodetool upgradesstables 
> 
> So, must we run upgradesstables or an offline scrub? 
> 
> Thanks in advance for your help, 
> 
> Romain.

C* 1.0.6 to 1.1.12: upgradesstables or scrub?

[Romain HARDOUIN]

Hi all,

We are migrating from C* 1.0.6 to 1.1.12 and after reading DataStax 
documentation (http://www.datastax.com/docs/1.1/install/upgrading) we are 
a bit confused. 
The section "Completing the Upgrade", step 6, points out:

If you are upgrading from Cassandra 1.1.3 or earlier to Cassandra 
1.1.5 or later, skip steps 7 and 8 of this procedure and go to Completing 
the upgrade from Cassandra 1.1.3 or earlier to Cassandra 1.1.5 or later. 
--> i.e. run an offline scrub

However according to the step 8:

After upgrading, run nodetool upgradesstables against each node before 
running repair, moving nodes, or adding new ones. 
If you are using Cassandra 1.0.3 and earlier, use nodetool scrub 
instead of nodetool upgradesstables

So, must we run upgradesstables or an offline scrub?
 
Thanks in advance for your help,

Romain.

Re: Cassandra HANGS after some writes

[Naresh Yadav]

Hi Alex,

Yes i am testing in development environment of Windows 7 64bit.
I left default yaml then cassandra created "var" folder and created data,
log, cache folders in it.I tried commit log on different harddisk but
this problem not solved with thatI guess this problem is somewhat
related to deadlock in Native Transport threads...thats why cassandra is
hanging indefinitly..

Naresh

On Tue, Aug 13, 2013 at 7:21 PM, Alexis Rodríguez <
arodrig...@inconcertcc.com> wrote:

> Naresh, are you deploying cassandra in windows?
>
> If that is the case you may need to change the data and commitlog
> directories in cassandra.yaml. Also you should check the log directories.
>
> See the section 2.1  http://wiki.apache.org/cassandra/GettingStarted
>
>
> On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav wrote:
>
>>
>> Thanks Alain, will avoid capsi am newbie to cassandra, just started
>> using 2 weeks back..
>>
>> Here are JConsole screenshots just 5mins after cassandra freezed :
>>
>> http://i.imgur.com/3oUBjKU.png
>> http://i.imgur.com/2O4PrKb.png
>> http://i.imgur.com/zxhFzr1.png   4:05 is time cassandra
>> freezed thats why decline in no of threads
>> http://i.imgur.com/ScgAciv.png
>> Uploaded complete system.log of cassandra till freeze :
>> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>>
>> Observation : As in my usecase i am storing 1lakh
>> combinations(527insert,506update,954select) each parallel by 100 threads in
>> batch of 1000...
>> Sometimes it works till 1000 batch then hangs but sometimes it completes
>> 1 then hangs and once even worked for more than lakh
>> Same hardware Same settings of cassandra i see random behaviour of
>> performance..
>>
>> Thanks
>> Naresh
>>
>>
>> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ wrote:
>>
>>> Hi Naresh.
>>>
>>> First thing, there is no need of caps in here. People reading this ML is
>>> here to help when they have time and skills enough to do so. So please,
>>> chill out and do not use caps to show how much desperate you are.
>>>
>>> Concerning your problem, the only abnormal thing I was able to find in
>>> your logs is
>>>
>>>
>>>1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>>>SSTableDeletingTask.java (line 72) Unable to delete
>>>
>>> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>>>(it will be removed on server restart; we'll also retry after GC)
>>>
>>>
>>> I don't think this should keep C* hanging.
>>>
>>> Do you have something on kernel logs ?
>>>
>>> Do you have monitor any metrics like disk throughput / heap used / cpu
>>> load / iowait which are known as being bottlenecks / pertinent metrics ?
>>>
>>> Alain
>>>
>>>
>>> 2013/8/13 Naresh Yadav 
>>>

 Hi All,

 I have single node cassandra using  CQL using datastax java driver
 1.0.1 and cassandra verison 1.2.6.

 *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
 processor.. DEFAULT cassandra.yaml no change done by me.
 -Xms1G^
  -Xmx12G^ no other change in cassandra.bat

 *Problem :
 _
  *cassandra Freezes after some writes and i see no action on cassandra
 console for an hour...all Native_Transport threads are also killedmy
 program keeps running NO ERROR comes...when i connect with cql that
 works In start it creates 16 NativeTransport threads and after
 10-15 minutes Total threads goes to 128...Just before it hangs, With
 JCONSOLE when i see Native_Transport threads then i find most of them in
 state as :

 http://pastebin.com/DeShpHtP

 *Load on cassandra : *
 ___
 i am running a usecase which stores Combinations(my project
 terminology) in cassandraCurrently testing storing 2.5 lakh
 combinations with 100 parallel threads..each thread storing one
 combination...real case i need to support of many CRORES but that would
 need different hardware and multi node cluster...

 In Storing ONE combination takes around 2sec and involves :

 527 INSERT INTO queries 

 506 UPDATE queries 

 954 SELECT queries 

 100 parallel threads parallely storing 100 combinations
 *MY CASSANDRA LOGS :*

 http://pastebin.com/CnNvA9x3

 Please look last 100-200 lines of log because that is time it freezed


 PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...

>>>
>>>
>>
>>
>>
>

Re: Cassandra HANGS after some writes

[Alexis Rodríguez]

Naresh, are you deploying cassandra in windows?

If that is the case you may need to change the data and commitlog
directories in cassandra.yaml. Also you should check the log directories.

See the section 2.1  http://wiki.apache.org/cassandra/GettingStarted


On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav  wrote:

>
> Thanks Alain, will avoid capsi am newbie to cassandra, just started
> using 2 weeks back..
>
> Here are JConsole screenshots just 5mins after cassandra freezed :
>
> http://i.imgur.com/3oUBjKU.png
> http://i.imgur.com/2O4PrKb.png
> http://i.imgur.com/zxhFzr1.png   4:05 is time cassandra
> freezed thats why decline in no of threads
> http://i.imgur.com/ScgAciv.png
> Uploaded complete system.log of cassandra till freeze :
> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>
> Observation : As in my usecase i am storing 1lakh
> combinations(527insert,506update,954select) each parallel by 100 threads in
> batch of 1000...
> Sometimes it works till 1000 batch then hangs but sometimes it completes
> 1 then hangs and once even worked for more than lakh
> Same hardware Same settings of cassandra i see random behaviour of
> performance..
>
> Thanks
> Naresh
>
>
> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ wrote:
>
>> Hi Naresh.
>>
>> First thing, there is no need of caps in here. People reading this ML is
>> here to help when they have time and skills enough to do so. So please,
>> chill out and do not use caps to show how much desperate you are.
>>
>> Concerning your problem, the only abnormal thing I was able to find in
>> your logs is
>>
>>
>>1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>>SSTableDeletingTask.java (line 72) Unable to delete
>>
>> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>>(it will be removed on server restart; we'll also retry after GC)
>>
>>
>> I don't think this should keep C* hanging.
>>
>> Do you have something on kernel logs ?
>>
>> Do you have monitor any metrics like disk throughput / heap used / cpu
>> load / iowait which are known as being bottlenecks / pertinent metrics ?
>>
>> Alain
>>
>>
>> 2013/8/13 Naresh Yadav 
>>
>>>
>>> Hi All,
>>>
>>> I have single node cassandra using  CQL using datastax java driver 1.0.1
>>> and cassandra verison 1.2.6.
>>>
>>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>>> processor.. DEFAULT cassandra.yaml no change done by me.
>>> -Xms1G^
>>>  -Xmx12G^ no other change in cassandra.bat
>>>
>>> *Problem :
>>> _
>>>  *cassandra Freezes after some writes and i see no action on cassandra
>>> console for an hour...all Native_Transport threads are also killedmy
>>> program keeps running NO ERROR comes...when i connect with cql that
>>> works In start it creates 16 NativeTransport threads and after
>>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>>> JCONSOLE when i see Native_Transport threads then i find most of them in
>>> state as :
>>>
>>> http://pastebin.com/DeShpHtP
>>>
>>> *Load on cassandra : *
>>> ___
>>> i am running a usecase which stores Combinations(my project terminology)
>>> in cassandraCurrently testing storing 2.5 lakh combinations with 100
>>> parallel threads..each thread storing one combination...real case i need to
>>> support of many CRORES but that would need different hardware and multi
>>> node cluster...
>>>
>>> In Storing ONE combination takes around 2sec and involves :
>>>
>>> 527 INSERT INTO queries 
>>>
>>> 506 UPDATE queries 
>>>
>>> 954 SELECT queries 
>>>
>>> 100 parallel threads parallely storing 100 combinations
>>> *MY CASSANDRA LOGS :*
>>>
>>> http://pastebin.com/CnNvA9x3
>>>
>>> Please look last 100-200 lines of log because that is time it freezed
>>>
>>>
>>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>>
>>
>>
>
>
>

Re: Cassandra HANGS after some writes

[Naresh Yadav]

Thanks Alain, will avoid capsi am newbie to cassandra, just started
using 2 weeks back..

Here are JConsole screenshots just 5mins after cassandra freezed :

http://i.imgur.com/3oUBjKU.png
http://i.imgur.com/2O4PrKb.png
http://i.imgur.com/zxhFzr1.png   4:05 is time cassandra freezed
thats why decline in no of threads
http://i.imgur.com/ScgAciv.png
Uploaded complete system.log of cassandra till freeze :
http://www.scribd.com/doc/159949231/Cassandrasystem-log

Observation : As in my usecase i am storing 1lakh
combinations(527insert,506update,954select) each parallel by 100 threads in
batch of 1000...
Sometimes it works till 1000 batch then hangs but sometimes it completes
1 then hangs and once even worked for more than lakh
Same hardware Same settings of cassandra i see random behaviour of
performance..

Thanks
Naresh

On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ  wrote:

> Hi Naresh.
>
> First thing, there is no need of caps in here. People reading this ML is
> here to help when they have time and skills enough to do so. So please,
> chill out and do not use caps to show how much desperate you are.
>
> Concerning your problem, the only abnormal thing I was able to find in
> your logs is
>
>
>1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>SSTableDeletingTask.java (line 72) Unable to delete
>
> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>(it will be removed on server restart; we'll also retry after GC)
>
>
> I don't think this should keep C* hanging.
>
> Do you have something on kernel logs ?
>
> Do you have monitor any metrics like disk throughput / heap used / cpu
> load / iowait which are known as being bottlenecks / pertinent metrics ?
>
> Alain
>
>
> 2013/8/13 Naresh Yadav 
>
>>
>> Hi All,
>>
>> I have single node cassandra using  CQL using datastax java driver 1.0.1
>> and cassandra verison 1.2.6.
>>
>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>> processor.. DEFAULT cassandra.yaml no change done by me.
>> -Xms1G^
>>  -Xmx12G^ no other change in cassandra.bat
>>
>> *Problem :
>> _
>>  *cassandra Freezes after some writes and i see no action on cassandra
>> console for an hour...all Native_Transport threads are also killedmy
>> program keeps running NO ERROR comes...when i connect with cql that
>> works In start it creates 16 NativeTransport threads and after
>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>> JCONSOLE when i see Native_Transport threads then i find most of them in
>> state as :
>>
>> http://pastebin.com/DeShpHtP
>>
>> *Load on cassandra : *
>> ___
>> i am running a usecase which stores Combinations(my project terminology)
>> in cassandraCurrently testing storing 2.5 lakh combinations with 100
>> parallel threads..each thread storing one combination...real case i need to
>> support of many CRORES but that would need different hardware and multi
>> node cluster...
>>
>> In Storing ONE combination takes around 2sec and involves :
>>
>> 527 INSERT INTO queries 
>>
>> 506 UPDATE queries 
>>
>> 954 SELECT queries 
>>
>> 100 parallel threads parallely storing 100 combinations
>> *MY CASSANDRA LOGS :*
>>
>> http://pastebin.com/CnNvA9x3
>>
>> Please look last 100-200 lines of log because that is time it freezed
>>
>>
>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>
>
>

Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?

[Richard Low]

On 13 August 2013 10:15, Alain RODRIGUEZ  wrote:

Streaming from all the physical nodes in the cluster should make repair
> faster, for the same reason it makes bootstrap faster. Shouldn't it ?
>

Virtual nodes doesn't speed up either very much.  Repair and bootstrap will
be limited by the node doing repair or bootstrap, since it has to do the
same amount of work whatever num_tokens is.  It places a more even load
across the rest of the cluster though, since it will repair with or
bootstrap from all nodes in the cluster.  So the overall time will in most
cases be about the same.

The real speedup from vnodes comes when running removenode, when the
streaming happens in parallel across all nodes.

Richard.

Re: Cassandra HANGS after some writes

[Alain RODRIGUEZ]

Hi Naresh.

First thing, there is no need of caps in here. People reading this ML is
here to help when they have time and skills enough to do so. So please,
chill out and do not use caps to show how much desperate you are.

Concerning your problem, the only abnormal thing I was able to find in your
logs is


   1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
   SSTableDeletingTask.java (line 72) Unable to delete
   
\var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
   (it will be removed on server restart; we'll also retry after GC)


I don't think this should keep C* hanging.

Do you have something on kernel logs ?

Do you have monitor any metrics like disk throughput / heap used / cpu load
/ iowait which are known as being bottlenecks / pertinent metrics ?

Alain


2013/8/13 Naresh Yadav 

>
> Hi All,
>
> I have single node cassandra using  CQL using datastax java driver 1.0.1
> and cassandra verison 1.2.6.
>
> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
> processor.. DEFAULT cassandra.yaml no change done by me.
> -Xms1G^
>  -Xmx12G^ no other change in cassandra.bat
>
> *Problem :
> _
>  *cassandra Freezes after some writes and i see no action on cassandra
> console for an hour...all Native_Transport threads are also killedmy
> program keeps running NO ERROR comes...when i connect with cql that
> works In start it creates 16 NativeTransport threads and after
> 10-15 minutes Total threads goes to 128...Just before it hangs, With
> JCONSOLE when i see Native_Transport threads then i find most of them in
> state as :
>
> http://pastebin.com/DeShpHtP
>
> *Load on cassandra : *
> ___
> i am running a usecase which stores Combinations(my project terminology)
> in cassandraCurrently testing storing 2.5 lakh combinations with 100
> parallel threads..each thread storing one combination...real case i need to
> support of many CRORES but that would need different hardware and multi
> node cluster...
>
> In Storing ONE combination takes around 2sec and involves :
>
> 527 INSERT INTO queries 
>
> 506 UPDATE queries 
>
> 954 SELECT queries 
>
> 100 parallel threads parallely storing 100 combinations
> *MY CASSANDRA LOGS :*
>
> http://pastebin.com/CnNvA9x3
>
> Please look last 100-200 lines of log because that is time it freezed
>
>
> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>

Cassandra HANGS after some writes

[Naresh Yadav]

Hi All,

I have single node cassandra using  CQL using datastax java driver 1.0.1
and cassandra verison 1.2.6.

*Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
processor.. DEFAULT cassandra.yaml no change done by me.
-Xms1G^
 -Xmx12G^ no other change in cassandra.bat

*Problem :
_
 *cassandra Freezes after some writes and i see no action on cassandra
console for an hour...all Native_Transport threads are also killedmy
program keeps running NO ERROR comes...when i connect with cql that
works In start it creates 16 NativeTransport threads and after
10-15 minutes Total threads goes to 128...Just before it hangs, With
JCONSOLE when i see Native_Transport threads then i find most of them in
state as :

http://pastebin.com/DeShpHtP

*Load on cassandra : *
___
i am running a usecase which stores Combinations(my project terminology) in
cassandraCurrently testing storing 2.5 lakh combinations with 100
parallel threads..each thread storing one combination...real case i need to
support of many CRORES but that would need different hardware and multi
node cluster...

In Storing ONE combination takes around 2sec and involves :

527 INSERT INTO queries 

506 UPDATE queries 

954 SELECT queries 

100 parallel threads parallely storing 100 combinations
*MY CASSANDRA LOGS :*

http://pastebin.com/CnNvA9x3

Please look last 100-200 lines of log because that is time it freezed


PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...

Re: cassandra 1.2.5- virtual nodes (num_token) pros/cons?

[Alain RODRIGUEZ]

@Aaron

Streaming from all the physical nodes in the cluster should make repair
faster, for the same reason it makes bootstrap faster. Shouldn't it ?




2013/8/12 Aaron Morton 

> Aaron - I read about the virtual nodes at
> http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2
>
>
> Thanks, I did not see anything in there about making repair smoother /
> faster.
>
> Cheers
> A
>  -
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 10/08/2013, at 11:39 AM, rash aroskar  wrote:
>
> Aaron - I read about the virtual nodes at
> http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2
>
>
>
>
>
>
>
>
> On Tue, Aug 6, 2013 at 4:49 AM, Richard Low  wrote:
>
>> On 6 August 2013 08:40, Aaron Morton  wrote:
>>
>>> The reason for me looking at virtual nodes is because of terrible
>>> experiences we had with 0.8 repairs and as per documentation (an logically)
>>> the virtual nodes seems like it will help repairs being smoother. Is this
>>> true?
>>>
>>> I've not thought too much about how they help repair run smoother, what
>>> was the documentation you read ?
>>>
>>
>> There might be a slight improvement but I haven't observed any.  The
>> difference might be that, because every node shares replicas with every
>> other (with high probability), a single repair operation does the same work
>> on the node it was called on, but the rest is spread out over the cluster,
>> rather than just the RF nodes either side of the repairing node.  This
>> means the post-repair compaction work will take less time and the length of
>> time a node is loaded for during repair is less.
>>
>> However, the other benefits of vnodes are likely to be much more useful.
>>
>> Richard.
>>
>
>
>

Re: understanding memory footprint

[Alain RODRIGUEZ]

if using 1.2.*, Bloom filters are in native memory so not pressuring your
heap, how many data do you have per node ? If this value is big, you have
samples index in the heap consuming a lot of memory, for sure, and growing
as your data per node grow.

Solutions : increase the heap if < 8GB and / or reduce
sampling index_interval: 128 to a bigger value (256 - 512) and /or wait for
2.0.* which, of the top of my head, should move the sampling in native
memory allowing heap size to be independent from the data size per node.

This should alleviate things. Yet these are only guesses since I know
almost nothing about your cluster...

Hope this help somehow.


2013/8/12 Robert Coli 

> On Mon, Aug 12, 2013 at 11:14 AM, Paul Ingalls wrote:
>
>> I don't really need exact numbers, just a rough cost would be sufficient.
>>  I'm running into memory problems on my cluster, and I'm trying to decide
>> if reducing the number of column families would be worth the effort.
>>  Looking at the rule of thumb from the wiki entry made it seem like
>> reducing the number of tables would make a big impact, but I'm running
>> 1.2.8 so not sure if it is still true.
>>
>> Is there a new rule of thumb?
>>
>
> If you want a cheap/quick measure of how much space partially full
> memtables are taking, just nodetool flush and check heap usage before and
> after?
>
> If you want a cheap/quick measure of how much space empty sstables take in
> heap, I think you're out of luck.
>
> =Rob
>
>