Re: Best strategy for adding new nodes to the cluster
On 28 Sep 2010, at 08:37, Michael Dürgner wrote: >> What do you mean by "running live"? I am also planning to use cassandra on >> EC2 using small nodes. Small nodes have 1/4 cpu of the large ones, 1/4 cost, >> but I/O is more than 1/4 (amazon does not give explicit I/O numbers...), so >> I think 4 small instances should perform better than 1 large one (and the >> cost is the same), am I wrong? > > Based on results we saw and what you also find in different sources around > the web, EC2 small instances perform worse than 1/4 regarding IO performance. Ditto. My tests indicate that while the peak IO performance of small nodes can be ok (up to 1/2 of large), it degrades over time down to 1/6 or even less. It seems that Amazon dedicates sufficient bandwidth to small nodes in the beginning to ensure a smooth and quick boot, but then throttles down fairly aggressively within a few minutes. This seems to affect reads more than writes, though. Note also that large instances have over 4x the memory (1.7 GB => 7.5 GB), and that makes a world of difference (you can have larger caches, for example). You don't really want to start swapping on the small instances. (However, small instances are awesome for doing testing and learning how to manage a cluster.) /Janne
Re: Is there a debian 0.6.1 install package archived anywhere?
On Mon, 2010-09-27 at 14:42 -0700, Kyusik Chung wrote: > We're running a cassandra cluster using 0.6.1 and we need to add a > node. Id like to add another node with the same version so that I > don't have to test a mixed version cluster or test (and conduct) a > whole-scale upgrade to 0.6.5and I'd like to use the debian package > to install. Does anyone know of a copy of the 0.6.1 debian package > that might still be available? http://archive.apache.org/dist/cassandra/debian/pool/main/c/cassandra/cassandra_0.6.1_all.deb -- Eric Evans eev...@rackspace.com
Migrating from Mysql to Cassandra
Hi I am involving in migrating our database Mysql to Cassandra. After Google about Cassandra database i feel following are some issues we need to find the solutions. Suggest me if my analysis is wrong or for my doubts. *1) Projecting all the rows* - If we fixed KeyRange count to N records and initially start key & end key = "", then after first iteration of values last key as next start key and end key = "" it is going infinite loop. * 2) Getting rows based on some condition ( created date b/n date1 to date2) - *For one column other than actual key in ColumnFamily i found some solution in the following link. http://maxgrinev.com/2010/07/12/do-you-really-need-sql-to-do-it-all-in-cassandra/- select section, so we will have 2 ColumnFamilies. In our scenario we have to get records more than one column ( some times it will change periodically). So how many ColumnFamilies need to design, if requirements need to change in middle of the devlopment/ production means how to handle. * * * 3) Deleted keys are available in ColumnFamilies *- How to remove these keys. 4) If we delete a record in clustered environment (In a single node), after that before sync with remaining nodes the node got failure means we wont have the data consistency - how to handle it. This database wont follow ACID properties, how to handle this one for a huge ERP kind of application. 5) Suggest me some GUI clients for Cassandra Database - to cross check the records in backend. /Regards Sekhar. * * * *
Re: Dazed and confused with Cassandra on EC2 ...
Peter - my apologies for the slow response - we had to divert down a 'Plan B' approach last week involving MySQL, memcache, redis and various other uglies. On 20 September 2010 23:11, Peter Schuller wrote: > Are you running an old JVM by any chance? (Just grasping for straws.) JVM is Sun's 1.6 - I've been caught out once before with openjdk's performance challenges, so I'm particularly careful with this now. > Hmm. I can see useless spinning decreasing efficiency, but the numbers > from your log are really extreme. Do you have a URL / bug id or > anything that one can read up on about this? We've rebuilt the Cassandra cluster this week, avoiding Hadoop entirely - partly to reduce the variables in play, and partly because it looks like we'll only need two 'feeder' nodes for our jobs with the size of Cassandra cluster that we're likely going to end up with (10-12 ish). Any ratio higher than that seems to, on EC2 at least, cause too many fails on the Cassandra side. Actually, there's a question - is it 'acceptable' do you think for GC to take out a small number of your nodes at a time, so long as the bulk (or at least where RF is > nodes gone on STW GC) of the nodes are okay? I suspect this is a question peculiar to Amazon EC2, as I've never seen a box rendered non-communicative by a single core flat-lining. By the end of this week we hope to have a better idea (mind, I've thought that for the past 5 weeks of experimenting). If I'm back to square one at that point I'll start pastebining some logs and configs. Increasingly, I'm convinced that many of these problems would be solved if we hosted our own servers. cheers, Jedd.
Schema Questions?
Question regarding schema set-up and modification: I just set up the newest nightly build of 0.7, imported our .yaml file after starting Cassandra, and everything is honky-dorey. We're not sure if Pelops is capable of modifying shema definitions, but it doesn't appear any changes we make to the .yaml file after the initial import go through (re-importing using schematool doesn't work). How could I change, for example, the rows_cached or name fields below on the fly without losing data? Is this possible? - name: Example column_type: Super compare_with: UTF8Type gc_grace_seconds: 864000 keys_cached: 20.0 preload_row_cache: false read_repair_chance: 1.0 rows_cached: 1.0 Frank
Re: Schema Questions?
> Question regarding schema set-up and modification: I just set up the newest > nightly build of 0.7, imported our .yaml file after starting Cassandra, and > everything is honky-dorey. We're not sure if Pelops is capable of modifying > shema definitions, but it doesn't appear any changes we make to the .yaml > file after the initial import go through (re-importing using schematool > doesn't work). Expected in 0.7 where the configuration file is essentially just an example: http://wiki.apache.org/cassandra/FAQ#no_keyspaces http://wiki.apache.org/cassandra/LiveSchemaUpdates > How could I change, for example, the rows_cached or name > fields below on the fly without losing data? Is this possible? Check out the system_* methods at the bottom of interface/cassandra.thrift. I believe these are in working order for 0.7. -- / Peter Schuller
Re: Schema Questions?
I understand adding column family configuration fields using thrift in Pelops for system_add_column_family, and even renaming column families, but what about modifying the actual column family configuration values, e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)? On Tue, Sep 28, 2010 at 1:17 PM, Peter Schuller wrote: > > Question regarding schema set-up and modification: I just set up the > newest > > nightly build of 0.7, imported our .yaml file after starting Cassandra, > and > > everything is honky-dorey. We're not sure if Pelops is capable of > modifying > > shema definitions, but it doesn't appear any changes we make to the .yaml > > file after the initial import go through (re-importing using schematool > > doesn't work). > > Expected in 0.7 where the configuration file is essentially just an > example: > > http://wiki.apache.org/cassandra/FAQ#no_keyspaces > http://wiki.apache.org/cassandra/LiveSchemaUpdates > > > How could I change, for example, the rows_cached or name > > fields below on the fly without losing data? Is this possible? > > Check out the system_* methods at the bottom of > interface/cassandra.thrift. I believe these are in working order for > 0.7. > > -- > / Peter Schuller >
Re: Migrating from Mysql to Cassandra
1) Thats the general approach http://wiki.apache.org/cassandra/FAQ#iter_world for Not sure why you think it will run into an infinite looop.2) Not sure what your question is here. You either need to maintain the indexes yourself (by using a CF where the cols are the keys for another CF) or use the secondary indexes coming in v 0.7. *But* you should experiment with different ways to donormalise your data to remove the need for indexes. The best design is where you make a single request to a single CF. 3) See the FAQ http://wiki.apache.org/cassandra/FAQ#range_ghosts When doing a get_range_slices it's best to test the number of columns returned for each row to see if it has what you expected. 4) Not sure what the question is, perhaps this section on Consistency will help http://wiki.apache.org/cassandra/ArchitectureOverview5) GUI clients http://wiki.apache.org/cassandra/FAQ#guiThere is a lot of information in the Wiki. AaronOn 29 Sep, 2010,at 04:32 AM, sekhar kosuru wrote:Hi I am involving in migrating our database Mysql to Cassandra. After Google about Cassandra database i feel following are some issues we need to find the solutions.Suggest me if my analysis is wrong or for my doubts. 1) Projecting all the rows - If we fixed KeyRange count to N records and initially start key & end key = "", then after first iteration of values last key as next start key and end key = "" it is going infinite loop. 2) Getting rows based on some condition ( created date b/n date1 to date2) - For one column other than actual key in ColumnFamily i found some solution in the following link. http://maxgrinev.com/2010/07/12/do-you-really-need-sql-to-do-it-all-in-cassandra/ - select section, so we will have 2 ColumnFamilies. In our scenario we have to get records more than one column ( some times it will change periodically). So how many ColumnFamilies need to design, if requirements need to change in middle of the devlopment/ production means how to handle. 3) Deleted keys are available in ColumnFamilies - How to remove these keys. 4) If we delete a record in clustered environment (In a single node), after that before sync with remaining nodes the node got failure means we wont have the data consistency - how to handle it. This database wont follow ACID properties, how to handle this one for a huge ERP kind of application. 5) Suggest me some GUI clients for Cassandra Database - to cross check the records in backend. /RegardsSekhar.
Re: Schema Questions?
There is a system_update_column_family(CDef) function on the trunk, so it will be in 0.7-beta2. Take a look at the interface/cassandra.thrift .AOn 29 Sep, 2010,at 09:31 AM, Frank LoVecchio wrote:I understand adding column family configuration fields using thrift in Pelops for system_add_column_family, and even renaming column families, but what about modifying the actual column family configuration values, e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)? On Tue, Sep 28, 2010 at 1:17 PM, Peter Schullerwrote: > Question regarding schema set-up and modification: I just set up the newest > nightly build of 0.7, imported our .yaml file after starting Cassandra, and > everything is honky-dorey. We're not sure if Pelops is capable of modifying > shema definitions, but it doesn't appear any changes we make to the .yaml > file after the initial import go through (re-importing using schematool > doesn't work). Expected in 0.7 where the configuration file is essentially just an example: http://wiki.apache.org/cassandra/FAQ#no_keyspaces http://wiki.apache.org/cassandra/LiveSchemaUpdates > How could I change, for example, the rows_cached or name > fields below on the fly without losing data? Is this possible? Check out the system_* methods at the bottom of interface/cassandra.thrift. I believe these are in working order for 0.7 -- / Peter Schuller
Re: Schema Questions?
Ok, I tried that method in Pelops, and this is the error I get when trying to to update a Column Family Type from Standard to Super: InvalidRequestException(why:types do not match.) at org.apache.cassandra.thrift.Cassandra$system_update_column_family_result.read(Cassandra.java:27741) at org.apache.cassandra.thrift.Cassandra$Client.recv_system_update_column_family(Cassandra.java:1579) at org.apache.cassandra.thrift.Cassandra$Client.system_update_column_family(Cassandra.java:1554) I have last night's build of Cassandra. On Tue, Sep 28, 2010 at 2:55 PM, Aaron Morton wrote: > There is a system_update_column_family(CDef) function on the trunk, so it > will be in 0.7-beta2. Take a look at the interface/cassandra.thrift . > > > A > > On 29 Sep, 2010,at 09:31 AM, Frank LoVecchio wrote: > > I understand adding column family configuration fields using thrift in > Pelops for system_add_column_family, and even renaming column families, but > what about modifying the actual column family configuration values, > e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)? > > > > On Tue, Sep 28, 2010 at 1:17 PM, Peter Schuller < > peter.schul...@infidyne.com> wrote: > >> > Question regarding schema set-up and modification: I just set up the >> newest >> > nightly build of 0.7, imported our .yaml file after starting Cassandra, >> and >> > everything is honky-dorey. We're not sure if Pelops is capable of >> modifying >> > shema definitions, but it doesn't appear any changes we make to the >> .yaml >> > file after the initial import go through (re-importing using schematool >> > doesn't work). >> >> Expected in 0.7 where the configuration file is essentially just an >> example: >> >> http://wiki.apache.org/cassandra/FAQ#no_keyspaces >> http://wiki.apache.org/cassandra/LiveSchemaUpdates >> >> > How could I change, for example, the rows_cached or name >> > fields below on the fly without losing data? Is this possible? >> >> Check out the system_* methods at the bottom of >> interface/cassandra.thrift. I believe these are in working order for >> 0.7 >> >> -- >> / Peter Schuller >> > >
Re: Schema Questions?
That looks like an error thrown from CFMetaData, telling you it's not possible to change a Standard CF to a Super CF. Which seems logical to me, how would it move the data from a standard CF to a super CF? You'll need to create a new Super CF and migrate the data manually. AaronOn 29 Sep, 2010,at 10:15 AM, Frank LoVecchio wrote:Ok, I tried that method in Pelops, and this is the error I get when trying to to update a Column Family Type from Standard to Super: InvalidRequestException(why:types do not match.) at org.apache.cassandra.thrift.Cassandra$system_update_column_family_result.read(Cassandra.java:27741) at org.apache.cassandra.thrift.Cassandra$Client.recv_system_update_column_family(Cassandra.java:1579) at org.apache.cassandra.thrift.Cassandra$Client.system_update_column_family(Cassandra.java:1554)I have last night's build of Cassandra. On Tue, Sep 28, 2010 at 2:55 PM, Aaron Mortonwrote: There is a system_update_column_family(CDef) function on the trunk, so it will be in 0.7-beta2. Take a look at the interface/cassandra.thrift .A On 29 Sep, 2010,at 09:31 AM, Frank LoVecchio wrote:I understand adding column family configuration fields using thrift in Pelops for system_add_column_family, and even renaming column families, but what about modifying the actual column family configuration values, e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)? On Tue, Sep 28, 2010 at 1:17 PM, Peter Schuller wrote: > Question regarding schema set-up and modification: I just set up the newest > nightly build of 0.7, imported our .yaml file after starting Cassandra, and > everything is honky-dorey. We're not sure if Pelops is capable of modifying > shema definitions, but it doesn't appear any changes we make to the .yaml > file after the initial import go through (re-importing using schematool > doesn't work). Expected in 0.7 where the configuration file is essentially just an example: http://wiki.apache.org/cassandra/FAQ#no_keyspaces http://wiki.apache.org/cassandra/LiveSchemaUpdates > How could I change, for example, the rows_cached or name > fields below on the fly without losing data? Is this possible? Check out the system_* methods at the bottom of interface/cassandra.thrift. I believe these are in working order for 0.7 -- / Peter Schuller
avro + cassandra + ruby
Hi, I'm attempting to use avro to talk to cassandra because the ruby thrift client's read performance is pretty bad (I measured 4x slower than java). However, I run into a problem when calling multiget_slice. The server gives a KeyspaceNotDefinedException because clientState.getKeyspace() returns null. It seems this is because ClientState stores the keyspace in a ThreadLocal. I call set_keyspace and clientState stores the keyspace value. I guess the next avro call to multiget_slice runs in a different thread so it can't retrieve the value. In ruby, I use Avro::IPC::HTTPTransceiver as the transport which I believe is a stateless transport. I also tried SocketTransport, but that died with a malloc exception. Is this a problem with the ruby avro library (I use avro 1.4.0), or how the server handles avro threads? Any help would be appreciated! Thanks, --Gabor
Re: avro + cassandra + ruby
On Tue, Sep 28, 2010 at 4:06 PM, Gabor Torok wrote: > Hi, > I'm attempting to use avro to talk to cassandra because the ruby thrift > client's read performance is pretty bad (I measured 4x slower than java). Only 4x feels like a win. :) One thing you should try is to make thrift use BinaryProtocolAccelerated, rather than the pure-ruby implementation (we should change the default). -ryan > However, I run into a problem when calling multiget_slice. > The server gives a KeyspaceNotDefinedException because > clientState.getKeyspace() returns null. > It seems this is because ClientState stores the keyspace in a ThreadLocal. > > I call set_keyspace and clientState stores the keyspace value. I guess the > next avro call to multiget_slice runs in a different thread so it can't > retrieve the value. > > In ruby, I use Avro::IPC::HTTPTransceiver as the transport which I believe is > a stateless transport. I also tried SocketTransport, but that died with a > malloc exception. > > Is this a problem with the ruby avro library (I use avro 1.4.0), or how the > server handles avro threads? > Any help would be appreciated! > > Thanks, > --Gabor >
Wed: Seattle Scalability / Hadoop / NoSQL Meetup: Killer Guests!
Hey guys! Just wanted to let you know that Wednesday's meetup is going to have some *fantastic* speakers from around the world :) You need to come! http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/calendar/13704368/ Wednesday, 7pm, Amazon SLU. First we have Tim Anglade, who flew here all the way from Paris, France. He's one of the foremost advocates of NoSQL, and the founder of "A NoSQL Summer". His talks are always thought-provoking, lovely to look at, and informative. He'll be talking about the lessons learned and mistakes of Hadoop and NoSQL, and what they'll look like in the future. We'll also have Ben Hendrickson from SEOMoz, who will talk about their home-built, fascinating distributed DB. IF you've ever wanted to know what it takes to build a database from scratch, you'll find this to be quite enlightening. And as always, there's delicious German beer and bratwurst at Fiererabend, 422 Yale Ave N., afterward. http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/calendar/13704368/ Cheers, Bradford -- Bradford Stephens, Founder, Drawn to Scale drawntoscalehq.com 727.697.7528 http://www.drawntoscalehq.com -- The intuitive, cloud-scale data solution. Process, store, query, search, and serve all your data. http://www.roadtofailure.com -- The Fringes of Scalability, Social Media, and Computer Science
Re: Is the secondary index maintained synchronously in 0.7
Why the secondary index is not updated absolutely atomically? 2010/9/2 Jonathan Ellis > yes, it is updated atomically (but not in isolation, it's possible for > a client to see changes to one but not the other temporarily) > > On Thu, Sep 2, 2010 at 1:47 PM, Alvin Jin wrote: > > > > Hello, > > > > I was thinking the details of the secondary index in 0.7. > > Will it be updated atomically with its base table? > > > > Any explaination the on secondary index is appreciated. > > Thanks. > > > > -- > > View this message in context: > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Is-the-secondary-index-maintained-synchronously-in-0-7-tp5492798p5492798.html > > Sent from the cassandra-u...@incubator.apache.org mailing list archive > at Nabble.com. > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: Is the secondary index maintained synchronously in 0.7
because it's not worth the penalty to concurrency On Tue, Sep 28, 2010 at 9:02 PM, Alvin UW wrote: > Why the secondary index is not updated absolutely atomically? > > 2010/9/2 Jonathan Ellis >> >> yes, it is updated atomically (but not in isolation, it's possible for >> a client to see changes to one but not the other temporarily) >> >> On Thu, Sep 2, 2010 at 1:47 PM, Alvin Jin wrote: >> > >> > Hello, >> > >> > I was thinking the details of the secondary index in 0.7. >> > Will it be updated atomically with its base table? >> > >> > Any explaination the on secondary index is appreciated. >> > Thanks. >> > >> > -- >> > View this message in context: >> > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Is-the-secondary-index-maintained-synchronously-in-0-7-tp5492798p5492798.html >> > Sent from the cassandra-u...@incubator.apache.org mailing list archive >> > at Nabble.com. >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Couple of minor logging / error message things
Noticed these when working against the current 0.7.0 beta2 (#3) build...When sending a system_add_keyspace request with an invalid keyspace the response to the client is fine... (python)InvalidRequestException: InvalidRequestException(why='Invalid keyspace name: Test Keyspace 1285729085.78')However it's not logged in the system.log. Looking through the code it appears that none of the Config errors are logged at any level. It may be handy to log these as a record of attempted changes on the system.In thrift/CassandraServer system_add_keyspace() the strategy_class string from the KsDef is used to load a class. The ClassNotFoundError is then caught and used to build an InvalidRequestException. If the strategy_class is missing or empty, the error returned to the client is (python)InvalidRequestException: InvalidRequestException(why='')Aaron
Re: avro + cassandra + ruby
Thanks, that made things better by about 30%. Unfortunately for me that's still unacceptable... :-( I feel like I'm doing something wrong with avro (see my original post). Was anyone able to make it work?
Re: Couple of minor logging / error message things
On Tue, Sep 28, 2010 at 10:28 PM, Aaron Morton wrote: > Noticed these when working against the current 0.7.0 beta2 (#3) build... > When sending a system_add_keyspace request with an invalid keyspace the > response to the client is fine... > > (python) > InvalidRequestException: InvalidRequestException(why='Invalid keyspace name: > Test Keyspace 1285729085.78') > However it's not logged in the system.log. We don't log any InvalidRequest. Not sure if it's worth special-casing system_ methods there. > In thrift/CassandraServer system_add_keyspace() the strategy_class string > from the KsDef is used to load a class. The ClassNotFoundError is then > caught and used to build an InvalidRequestException. If the strategy_class > is missing or empty, the error returned to the client is > (python) > InvalidRequestException: InvalidRequestException(why='') Can you create a ticket for this? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Truncate + Snapshot + Cannot Allocate Memory == Timeout
Running on the current 0.7.0 beta2 #3. On my dev workstation (unubtu 10.4) I accidentally let cassandra have 4GB and free memory got down to <100 MB . I was running some tests that added a few (< 100) rows to the CF's in Keyspace1, then Truncated the CF's for the next tests. I got the following error in the cassandra system.log DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,166 CassandraServer.java (line 954) truncating Standard1 in Keyspace1DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,166 StorageProxy.java (line 759) Starting a blocking truncate operation on keyspace Keyspace1, CF DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,166 StorageProxy.java (line 774) Starting to send truncate messages to hosts [localhost/127.0.0.1]DEBUG [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 TruncateVerbHandler.java (line 50) Applying Truncation(keyspace='Keyspace1', cf='Standard1')DEBUG [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 Tablejava (line 630) Truncating... INFO [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 ColumnFamilyStore.java (line 459) switching in a fresh Memtable for Standard1 at CommitLogContext(file='/local1/junkbox/cassandra/commitlog/CommitLog-1285728080403.log', position=10022050) INFO [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 ColumnFamilyStore.java (line 771) Enqueuing flush of memtable-standa...@1701009673(40104 bytes, 1152 operations)DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,167 StorageProxy.java (line 780) Sent all truncate messages, now waiting for 1 responses INFO [FLUSH-WRITER-POOL:1] 2010-09-29 16:44:39,167 Memtable.java (line 150) Writing memtable-standa...@1701009673(40104 bytes, 1152 operations) INFO [FLUSH-WRITER-POOL:1] 2010-09-29 16:44:39,268 Memtable.java (line 157) Completed flushing /local1/junkbox/cassandra/data/Keyspace1/Standard1-e-10-Data.dbDEBUG [CompactionExecutor:1] 2010-09-29 16:44:39,268 CompactionManager.java (line 95) Checking to see if compaction of Standard1 would be usefulDEBUG [COMMIT-LOG-WRITER] 2010-09-29 16:44:39,268 CommitLog.java (line 422) discard completed log segments for CommitLogContext(file='/local1/junkbox/cassandra/commitlog/CommitLog-1285728080403.log', position=10022050), column family 1000.DEBUG [COMMIT-LOG-WRITER] 2010-09-29 16:44:39,268 CommitLog.java (line 448) Marking replay position 10022050 on commit log CommitLogSegment(/local1/junkbox/cassandra/commitlog/CommitLog-1285728080403.log)ERROR [MUTATION_STAGE:25] 2010-09-29 16:44:39,341 AbstractCassandraDaemon.java (line 88) Fatal exception in thread Thread[MUTATION_STAGE:25,5,main]java.io.IOError: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1368) at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1511) at org.apache.cassandra.db.Table.truncate(Table.java:633) at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:54) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50) at javautil.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)Caused by: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) at org.apache.cassandra.io.util.FileUtils.createHardLinkWithExec(FileUtils.java:263) at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:229) at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1360) ... 7 moreCaused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory at java.lang.UNIXProcess.(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) ... 10 moreDEBUG [GC inspection] 2010-09-29 16:44:39,539 GCInspector.java (line 131) GC for ParNew: 43 ms, 29106728 reclaimed leaving 3567200784 used; max is 5498404864DEBUG [pool-1-thread-44] 2010-09-29 16:44:49,189 ClientState.java (line 107) logged out: #Fair enough, though it appeared as a timeout on the client (with a 10 second timeout) File "/tech/home//git_home/trojan/trojan/cassandra/Cassandrapy", line 846, in truncate self.recv_truncate() File "/tech/home//git_home/trojan/trojan/cassandra/Cassandra.py", line 857, in recv_truncate (fname, mtype, rseqid) = self._iprot.readMessageBegin() File "/tech/home//git_home/trojan/trojan/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin sz = self.readI32() chunk = self.read(sz-have) File "/tech/home//git_home/trojan/trojan/thrift/transport/TSocket.py", line 92, in read buff = self.handle.recv(sz)timeout: timed outHere's the output of ulimit -a core file size (bloc
Re: Truncate + Snapshot + Cannot Allocate Memory == Timeout
On Tue, Sep 28, 2010 at 11:25 PM, Aaron Morton wrote: > 1) Is the memory error just a result of me letting my machine run stupidly > low on memory? No, it's the JVM forking to run ln. Enable overcommit, or get JNA so it does the link w/ native code instead. > 2) Should it have returned an ApplicationError or some such in this case? > The code in ColumnFamilyStore:1368 is catching the IOException from the call > to FileUtils.createHardLink and wrapping it in an IOError. However the code > in TruncateVerbHandler:56 is looking for the IOException. That does sound like a bug. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: Is there a debian 0.6.1 install package archived anywhere?
Thanks Eric. Im not sure why I couldnt locate it earlierthought I had looked in all the directories. Kyusik Chung