Re: UnavailableException with 1 node down and RF=2?
Presumably the failure detector generated a false positive for a second node temporarily On Wed, Jun 30, 2010 at 10:55 PM, James Golick wrote: > Oops. I meant to say that I'm reading with CL.ONE. > > J. > > Sent from my iPhone. > > On 2010-07-01, at 1:39 AM, Benjamin Black wrote: > >> .QUORUM or .ALL (they are the same with RF=2). >> >> On Wed, Jun 30, 2010 at 10:22 PM, James Golick wrote: >>> 4 nodes, RF=2, 1 node down. >>> How can I get an UnavailableException in that scenario? >>> - J. > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: UnavailableException with 1 node down and RF=2?
Oops. I meant to say that I'm reading with CL.ONE. J. Sent from my iPhone. On 2010-07-01, at 1:39 AM, Benjamin Black wrote: > .QUORUM or .ALL (they are the same with RF=2). > > On Wed, Jun 30, 2010 at 10:22 PM, James Golick wrote: >> 4 nodes, RF=2, 1 node down. >> How can I get an UnavailableException in that scenario? >> - J.
Re: UnavailableException with 1 node down and RF=2?
.QUORUM or .ALL (they are the same with RF=2). On Wed, Jun 30, 2010 at 10:22 PM, James Golick wrote: > 4 nodes, RF=2, 1 node down. > How can I get an UnavailableException in that scenario? > - J.
UnavailableException with 1 node down and RF=2?
4 nodes, RF=2, 1 node down. How can I get an UnavailableException in that scenario? - J.
Re: RE: live nodes list in ring
problem solved with a full cluster stop and then start~ Thanks Stephen. But still wondering the reason 2010-07-01 发件人： Stephen Hamer 发送时间： 2010-07-01 08:39:13 收件人： user@cassandra.apache.org 抄送： 'huzhonghua'; 'GongJianTao（宫建涛）' 主题： RE: live nodes list in ring When this happens to me I have to do a full cluster restart. Even doing a rolling restart across the cluster doesn’t seem to fix them, all of the nodes need to be stopped at the same time. After bringing everything back up the ring is correct. Does anyone know how a cluster gets into this state? Stephen From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Wednesday, June 30, 2010 1:42 PM To: user@cassandra.apache.org Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' Subject: Re: live nodes list in ring At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
Re: Re: live nodes list in ring
To me, this happens all in a sudden. No nodes removed nor new nodes joined 2010-07-01 发件人： Benjamin Black 发送时间： 2010-07-01 09:55:59 收件人： user 抄送： 主题： Re: live nodes list in ring Does this happen after you have changed the ring topology, especially adding nodes? 2010/6/30 Stephen Hamer : > When this happens to me I have to do a full cluster restart. Even doing a > rolling restart across the cluster doesn't seem to fix them, all of the > nodes need to be stopped at the same time. After bringing everything back up > the ring is correct. > > > > Does anyone know how a cluster gets into this state? > > > > Stephen > > > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Wednesday, June 30, 2010 1:42 PM > To: user@cassandra.apache.org > Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' > Subject: Re: live nodes list in ring > > > > At start up do you see log lines like this > > > > Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster > > > > Are all the nodes listed? > > > > aaron > > On 30 Jun 2010, at 22:50, 王一锋 wrote: > > Hi, > > > > In a cassandra cluster, when issueing ring command on every nodes, some can > show all nodes in the cluster but some can only show some other nodes. > > All nodes share the same seed list. > > And even some of the nodes in the seed list have this problem. > > Restarting the problematic nodes won't solve it. > > Try closing firewalls with following commands > > > > service iptables stop > > > > Still won't work. > > > > Anyone got a clue? > > > > Thanks very much. > > > > Yifeng > >
Re: Re: live nodes list in ring
Yes, but not all nodes. Just the part that the restarting node can show on its ring. 2010-07-01 发件人： aaron morton 发送时间： 2010-07-01 04:42:56 收件人： user 抄送： 'huzhonghua'; 'GongJianTao（宫建涛 ）' 主题： Re: live nodes list in ring At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
Re: Implementing Counter on Cassandra
ZK is way overkill for counters. memcache and redis are much better at the job. On Tue, Jun 29, 2010 at 12:32 PM, Jonathan Shook wrote: > Until then, a pragmatic solution, however undesirable, would be to > only have a single logical thread/task/actor that is allowed to > read,modify,update. If this doesn't work for your application, then a > (distributed) lock manager may be used until such time that you can > take it out. Some are using ZooKeeper for this. > > > On Tue, Jun 29, 2010 at 11:45 AM, Ryan King wrote: >> On Tue, Jun 29, 2010 at 9:42 AM, Utku Can Topçu wrote: >>> Hey Guys, >>> >>> Currently in a project I'm involved in, I need to have some columns holding >>> incremented data. >>> The easy approach for implementing a counter with increments is right now as >>> I figured out is "read -> increment -> insert" however this approach is not >>> an atomic operation and can easily be corrupted in time. >>> >>> Do you have any best practices in implementing an atomic counter on >>> Cassandra? >> >> https://issues.apache.org/jira/browse/CASSANDRA-1072 >> >
Re: live nodes list in ring
Does this happen after you have changed the ring topology, especially adding nodes? 2010/6/30 Stephen Hamer : > When this happens to me I have to do a full cluster restart. Even doing a > rolling restart across the cluster doesn't seem to fix them, all of the > nodes need to be stopped at the same time. After bringing everything back up > the ring is correct. > > > > Does anyone know how a cluster gets into this state? > > > > Stephen > > > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Wednesday, June 30, 2010 1:42 PM > To: user@cassandra.apache.org > Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' > Subject: Re: live nodes list in ring > > > > At start up do you see log lines like this > > > > Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster > > > > Are all the nodes listed? > > > > aaron > > On 30 Jun 2010, at 22:50, 王一锋 wrote: > > Hi, > > > > In a cassandra cluster, when issueing ring command on every nodes, some can > show all nodes in the cluster but some can only show some other nodes. > > All nodes share the same seed list. > > And even some of the nodes in the seed list have this problem. > > Restarting the problematic nodes won't solve it. > > Try closing firewalls with following commands > > > > service iptables stop > > > > Still won't work. > > > > Anyone got a clue? > > > > Thanks very much. > > > > Yifeng > >
RE: live nodes list in ring
When this happens to me I have to do a full cluster restart. Even doing a rolling restart across the cluster doesn’t seem to fix them, all of the nodes need to be stopped at the same time. After bringing everything back up the ring is correct. Does anyone know how a cluster gets into this state? Stephen From: aaron morton [mailto:aa...@thelastpickle.com] Sent: Wednesday, June 30, 2010 1:42 PM To: user@cassandra.apache.org Cc: 'huzhonghua'; 'GongJianTao（宫建涛）' Subject: Re: live nodes list in ring At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
Re: Cluster-wide pause
use a larger heap or throttle your writes On Wed, Jun 30, 2010 at 12:12 PM, Steve Lihn wrote: > Jon, > I am experimenting writing 8 million rows into Cassandra and also > experienced some random timeouts, even with 10-second timeout parameter. > How do I avoid such timeout at all cost? (I.e. At this time, my priority is > to finish the end-to-end test. Don't want the program to fail at all.) > This is a one-node server with ms1G and mx2GB. The rest in cassandra.in.sh > are default I think. > > Steve > > On Wed, Jun 30, 2010 at 1:58 AM, Jonathan Ellis wrote: >> >> You could be seeing GC pauses. Did you increase the heap size you gave >> Cassandra, when you increased your VM size? >> >> On Tue, Jun 29, 2010 at 11:57 AM, Oren Benjamin >> wrote: >> > Hi all - first timer here. >> > >> > I'm experimenting with Cassandra on Rackspace Cloud. Started with 4GB >> > nodes and saw read latency spikes while streaming was taking place, so I >> > increased to 8GB to see if limited memory was the issue. Now I'm seeing >> > very strange behavior during any period that writes are taking place. The >> > entire (6 node) cluster seems to pause for periods of as much as 5-8 sec. >> > By that I mean all the stats (cpu, disk, network IO monitored via dstat) >> > drop to zero or near zero on all nodes simultaneously. Does anyone have >> > experience with Cassandra on Rackspace or any idea what's going on here? >> > >> > The pauses are short enough that it's difficult to introspect the >> > application and determine what it's doing during the pause, but long enough >> > to cause unacceptable latency for any service built on top of it. >> > >> > Any ideas or debugging methods would be greatly appreciated, >> > >> > -- Oren >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Cassandra Pig Integration
I am trying to query data using Pig 0.7.0, when i run contrib/pig/bin/pig_cassandra -x local filename.pig, i am getting an error ERROR 2117: Unexpected error when launching map reduce job. org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias topnames at org.apache.pig.PigServer.openIterator(PigServer.java:521) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:544) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) at org.apache.pig.Main.main(Main.java:391) Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias topnames at org.apache.pig.PigServer.store(PigServer.java:577) at org.apache.pig.PigServer.openIterator(PigServer.java:504) ... 6 more Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2117: Unexpected error when launching map reduce job. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:209) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:308) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:835) at org.apache.pig.PigServer.store(PigServer.java:569) ... 7 more Caused by: java.lang.RuntimeException: Could not resolve error that occured when launching map reduce job: java.lang.ExceptionInInitializerError at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$JobControlThreadExceptionHandler.uncaughtException(MapReduceLauncher.java:510) at java.lang.Thread.dispatchUncaughtException(Thread.java:1855) My pig file would look like rows = LOAD 'cassandra://Keyspace1/Standard1' USING CassandraStorage(); cols = FOREACH rows GENERATE flatten($1); colnames = FOREACH cols GENERATE $0; namegroups = GROUP colnames BY $0; namecounts = FOREACH namegroups GENERATE COUNT($1), group; orderednames = ORDER namecounts BY $0; topnames = LIMIT orderednames 50; dump topnames; Does anyone how to fix this issue? Thanks, Mubarak Seyed.
Re: live nodes list in ring
At start up do you see log lines like this Gossiper.java (line 576) Node /192.168.34.30 is now part of the cluster Are all the nodes listed? aaron On 30 Jun 2010, at 22:50, 王一锋 wrote: > Hi, > > In a cassandra cluster, when issueing ring command on every nodes, some can > show all nodes in the cluster but some can only show some other nodes. > All nodes share the same seed list. > And even some of the nodes in the seed list have this problem. > Restarting the problematic nodes won't solve it. > Try closing firewalls with following commands > > service iptables stop > > Still won't work. > > Anyone got a clue? > > Thanks very much. > > Yifeng
Re: simple authentication in cascal
Steve, Looking at the Cassandra docs it appears as though this shouldn't be too hard to implement, as it should simply involve adding a login method to the cascal session template. I'll make a ticket in github to track this , I hope that I have a little more time in the next few days to investigate further. Cheers. Chris On Wednesday, June 30, 2010, Steve Lihn wrote: > Chris, > What I have in mind is to use SimpleAuthenticator in conjuction with the > access.properties and passwd.properties. > This use case is actually somewhat personal. I have servers in the AWS cloud, > during development and testing, I may connect from home or wireless points on > the road. > I don't want to be bothered with opening up the firewall based on the IP I > have at that moment. So I am thinking to open up 9160 and use authentication > to block unauthorized writes, which may corrupt the database. But I couldn't > find any info on how to use SimpleAuthenticator in the scala client. > > Steve > > On Sat, Jun 26, 2010 at 3:07 PM, Chris Shorrock wrote: > > Hey Steve, > What type of authentication are you looking to do? Cassandra itself doesn't, > as far as I know, support any sort of user setup or authentication so it's on > the onus of the application developer to lock things down appropriately. > > > Chris > > On Fri, Jun 25, 2010 at 5:42 PM, Steve Lihn wrote: > > Searching how to use simple authentication in cascal. Couldn't find anything > in API docs. Does anybody know how to do it? > > Thanks, > Steve > > > >
Re: Compaction-pool pause -- was: Cluster-wide pause
Jon, Looks like this compaction process took 360 seconds. What is it and how do I avoid it? INFO [COMPACTION-POOL:1] 2010-06-30 14:56:08,667 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-129-Data.db'),org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-150-Data.db'),org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-172-Data.db'),org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-194-Data.db')] INFO [COMPACTION-POOL:1] 2010-06-30 15:02:09,537 CompactionManager.java (line 326) Compacted to /mnt/itops/cdata/TFO/CurrentHolding-197-Data.db. 1880884459/1880884459 bytes for 6306383 keys. Time: 360870ms. On Wed, Jun 30, 2010 at 3:12 PM, Steve Lihn wrote: > Jon, > I am experimenting writing 8 million rows into Cassandra and also > experienced some random timeouts, even with 10-second timeout parameter. > How do I avoid such timeout at all cost? (I.e. At this time, my priority is > to finish the end-to-end test. Don't want the program to fail at all.) > This is a one-node server with ms1G and mx2GB. The rest in cassandra.in.share > default I think. > > Steve > > >
Re: Cluster-wide pause
Jon, I am experimenting writing 8 million rows into Cassandra and also experienced some random timeouts, even with 10-second timeout parameter. How do I avoid such timeout at all cost? (I.e. At this time, my priority is to finish the end-to-end test. Don't want the program to fail at all.) This is a one-node server with ms1G and mx2GB. The rest in cassandra.in.share default I think. Steve On Wed, Jun 30, 2010 at 1:58 AM, Jonathan Ellis wrote: > You could be seeing GC pauses. Did you increase the heap size you gave > Cassandra, when you increased your VM size? > > On Tue, Jun 29, 2010 at 11:57 AM, Oren Benjamin > wrote: > > Hi all - first timer here. > > > > I'm experimenting with Cassandra on Rackspace Cloud. Started with 4GB > nodes and saw read latency spikes while streaming was taking place, so I > increased to 8GB to see if limited memory was the issue. Now I'm seeing > very strange behavior during any period that writes are taking place. The > entire (6 node) cluster seems to pause for periods of as much as 5-8 sec. > By that I mean all the stats (cpu, disk, network IO monitored via dstat) > drop to zero or near zero on all nodes simultaneously. Does anyone have > experience with Cassandra on Rackspace or any idea what's going on here? > > > > The pauses are short enough that it's difficult to introspect the > application and determine what it's doing during the pause, but long enough > to cause unacceptable latency for any service built on top of it. > > > > Any ideas or debugging methods would be greatly appreciated, > > > > -- Oren > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: simple authentication in cascal
Chris, What I have in mind is to use SimpleAuthenticator in conjuction with the access.properties and passwd.properties. This use case is actually somewhat personal. I have servers in the AWS cloud, during development and testing, I may connect from home or wireless points on the road. I don't want to be bothered with opening up the firewall based on the IP I have at that moment. So I am thinking to open up 9160 and use authentication to block unauthorized writes, which may corrupt the database. But I couldn't find any info on how to use SimpleAuthenticator in the scala client. Steve On Sat, Jun 26, 2010 at 3:07 PM, Chris Shorrock wrote: > Hey Steve, > > What type of authentication are you looking to do? Cassandra itself > doesn't, as far as I know, support any sort of user setup or authentication > so it's on the onus of the application developer to lock things down > appropriately. > > Chris > > > On Fri, Jun 25, 2010 at 5:42 PM, Steve Lihn wrote: > >> Searching how to use simple authentication in cascal. Couldn't find >> anything in API docs. Does anybody know how to do it? >> >> Thanks, >> Steve >> > >
live nodes list in ring
Hi, In a cassandra cluster, when issueing ring command on every nodes, some can show all nodes in the cluster but some can only show some other nodes. All nodes share the same seed list. And even some of the nodes in the seed list have this problem. Restarting the problematic nodes won't solve it. Try closing firewalls with following commands service iptables stop Still won't work. Anyone got a clue? Thanks very much. Yifeng
