Re: Best strategy for adding new nodes to the cluster

[Janne Jalkanen]

On 28 Sep 2010, at 08:37, Michael Dürgner wrote:

>> What do you mean by "running live"? I am also planning to use cassandra on 
>> EC2 using small nodes. Small nodes have 1/4 cpu of the large ones, 1/4 cost, 
>> but I/O is more than 1/4 (amazon does not give explicit I/O numbers...), so 
>> I think 4 small instances should perform better than 1 large one (and the 
>> cost is the same), am I wrong?
> 
> Based on results we saw and what you also find in different sources around 
> the web, EC2 small instances perform worse than 1/4 regarding IO performance.

Ditto. My tests indicate that while the peak IO performance of small nodes can 
be ok (up to 1/2 of large), it degrades over time down to 1/6 or even less. It 
seems that Amazon dedicates sufficient bandwidth to small nodes in the 
beginning to ensure a smooth and quick boot, but then throttles down fairly 
aggressively within a few minutes.  This seems to affect reads more than 
writes, though.

Note also that large instances have over 4x the memory (1.7 GB => 7.5 GB), and 
that makes a world of difference (you can have larger caches, for example). You 
don't really want to start swapping on the small instances.

(However, small instances are awesome for doing testing and learning how to 
manage a cluster.)

/Janne

Re: Is there a debian 0.6.1 install package archived anywhere?

[Eric Evans]

On Mon, 2010-09-27 at 14:42 -0700, Kyusik Chung wrote:
> We're running a cassandra cluster using 0.6.1 and we need to add a
> node.  Id like to add another node with the same version so that I
> don't have to test a mixed version cluster or test (and conduct) a
> whole-scale upgrade to 0.6.5and I'd like to use the debian package
> to install.  Does anyone know of a copy of the 0.6.1 debian package
> that might still be available?

http://archive.apache.org/dist/cassandra/debian/pool/main/c/cassandra/cassandra_0.6.1_all.deb

-- 
Eric Evans
eev...@rackspace.com

Migrating from Mysql to Cassandra

[sekhar kosuru]

Hi

I am involving in migrating our database Mysql to Cassandra. After Google
about Cassandra database i feel following are some issues we need to find
the solutions.

Suggest me if my analysis is wrong or for my doubts.

 *1) Projecting all the rows* - If we fixed KeyRange count to N records and
initially start key & end key = "", then after first iteration of values
last key as next start key and end key = "" it is going infinite loop.

*  2) Getting rows based on some condition ( created date b/n date1 to
date2)  -
  *For one column other than actual key in ColumnFamily i found some
solution in the following link.

http://maxgrinev.com/2010/07/12/do-you-really-need-sql-to-do-it-all-in-cassandra/-
select section, so we will have 2 ColumnFamilies. In our scenario we
have
to get records more than one column ( some times it will change
periodically).
So how many ColumnFamilies need to design, if requirements need to change in
middle of the devlopment/ production means how to handle.

*
*
*  3) Deleted keys are available in ColumnFamilies *- How to remove these
keys.

  4) If we delete a record in clustered environment (In a single node),
after that before sync with remaining nodes the node got failure means we
wont have the data consistency - how to handle it. This database wont follow
ACID properties, how to handle this one for a huge ERP kind of application.

 5) Suggest me some GUI clients for Cassandra Database - to cross check the
records in backend.


/Regards
Sekhar.

*
*
*
*

Re: Dazed and confused with Cassandra on EC2 ...

[Jedd Rashbrooke]

 Peter - my apologies for the slow response - we had
 to divert down a 'Plan B' approach last week involving
 MySQL, memcache, redis and various other uglies.

On 20 September 2010 23:11, Peter Schuller  wrote:
> Are you running an old JVM by any chance? (Just grasping for straws.)

 JVM is Sun's 1.6 - I've been caught out once before with
 openjdk's performance challenges, so I'm particularly
 careful with this now.

> Hmm. I can see useless spinning decreasing efficiency, but the numbers
> from your log are really extreme. Do you have a URL / bug id or
> anything that one can read up on about this?

 We've rebuilt the Cassandra cluster this week, avoiding
 Hadoop entirely - partly to reduce the variables in play,
 and partly because it looks like we'll only need two 'feeder'
 nodes for our jobs with the size of Cassandra cluster that
 we're likely going to end up with (10-12 ish).  Any ratio
 higher than that seems to, on EC2 at least, cause too many
 fails on the Cassandra side.

 Actually, there's a question - is it 'acceptable' do you think
 for GC to take out a small number of your nodes at a time,
 so long as the bulk (or at least where RF is > nodes gone
 on STW GC) of the nodes are okay?  I suspect this is a
 question peculiar to Amazon EC2, as I've never seen a box
 rendered non-communicative by a single core flat-lining.

 By the end of this week we hope to have a better idea (mind,
 I've thought that for the past 5 weeks of experimenting).  If I'm
 back to square one at that point I'll start pastebining some logs
 and configs.  Increasingly, I'm convinced that many of these
 problems would be solved if we hosted our own servers.

 cheers,
 Jedd.

Schema Questions?

[Frank LoVecchio]

Question regarding schema set-up and modification:  I just set up the newest
nightly build of 0.7, imported our .yaml file after starting Cassandra, and
everything is honky-dorey.  We're not sure if Pelops is capable of modifying
shema definitions, but it doesn't appear any changes we make to the .yaml
file after the initial import go through (re-importing using schematool
doesn't work). How could I change, for example, the rows_cached or name
fields below on the fly without losing data?  Is this possible?

- name: Example
  column_type: Super
  compare_with: UTF8Type
  gc_grace_seconds: 864000
  keys_cached: 20.0
  preload_row_cache: false
  read_repair_chance: 1.0
  rows_cached: 1.0

Frank

Re: Schema Questions?

[Peter Schuller]

> Question regarding schema set-up and modification:  I just set up the newest
> nightly build of 0.7, imported our .yaml file after starting Cassandra, and
> everything is honky-dorey.  We're not sure if Pelops is capable of modifying
> shema definitions, but it doesn't appear any changes we make to the .yaml
> file after the initial import go through (re-importing using schematool
> doesn't work).

Expected in 0.7 where the configuration file is essentially just an example:

   http://wiki.apache.org/cassandra/FAQ#no_keyspaces
   http://wiki.apache.org/cassandra/LiveSchemaUpdates

> How could I change, for example, the rows_cached or name
> fields below on the fly without losing data?  Is this possible?

Check out the system_* methods at the bottom of
interface/cassandra.thrift. I believe these are in working order for
0.7.

-- 
/ Peter Schuller

Re: Schema Questions?

[Frank LoVecchio]

I understand adding column family configuration fields using thrift in
Pelops for system_add_column_family, and even renaming column families, but
what about modifying the actual column family configuration values,
e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)?



On Tue, Sep 28, 2010 at 1:17 PM, Peter Schuller  wrote:

> > Question regarding schema set-up and modification:  I just set up the
> newest
> > nightly build of 0.7, imported our .yaml file after starting Cassandra,
> and
> > everything is honky-dorey.  We're not sure if Pelops is capable of
> modifying
> > shema definitions, but it doesn't appear any changes we make to the .yaml
> > file after the initial import go through (re-importing using schematool
> > doesn't work).
>
> Expected in 0.7 where the configuration file is essentially just an
> example:
>
>   http://wiki.apache.org/cassandra/FAQ#no_keyspaces
>   http://wiki.apache.org/cassandra/LiveSchemaUpdates
>
> > How could I change, for example, the rows_cached or name
> > fields below on the fly without losing data?  Is this possible?
>
> Check out the system_* methods at the bottom of
> interface/cassandra.thrift. I believe these are in working order for
> 0.7.
>
> --
> / Peter Schuller
>

Re: Migrating from Mysql to Cassandra

[Aaron Morton]

1) Thats the general approach http://wiki.apache.org/cassandra/FAQ#iter_world for Not sure why you think it will run into an infinite looop.2) Not sure what your question is here. You either need to maintain the indexes yourself (by using a CF where the cols are the keys for another CF) or use the secondary indexes coming in v 0.7. *But* you should experiment with different ways to donormalise your data to remove the need for indexes. The best design is where you make a single request to a single CF.  3) See the FAQ http://wiki.apache.org/cassandra/FAQ#range_ghosts When doing a get_range_slices it's best to test the number of columns returned for each row to see if it has what you expected. 4) Not sure what the question is, perhaps this section on Consistency will help http://wiki.apache.org/cassandra/ArchitectureOverview5) GUI clients http://wiki.apache.org/cassandra/FAQ#guiThere is a lot of information in the Wiki. AaronOn 29 Sep, 2010,at 04:32 AM, sekhar kosuru  wrote:Hi I am involving in migrating our database Mysql to Cassandra. After Google about Cassandra database i feel following are some issues we need to find the solutions.Suggest me if my analysis is wrong or for my doubts.
 1) Projecting all the rows - If we fixed KeyRange count to N records and initially start key & end key = "", then after first iteration of values last key as next start key and end key = "" it is going infinite loop.
  2) Getting rows based on some condition ( created date b/n date1 to date2)  -   For one column other than actual key in ColumnFamily i found some solution in the following link.    http://maxgrinev.com/2010/07/12/do-you-really-need-sql-to-do-it-all-in-cassandra/ - select section, so we will have 2 ColumnFamilies. In our scenario we have to get records more than one column ( some times it will change periodically).
So how many ColumnFamilies need to design, if requirements need to change in middle of the devlopment/ production means how to handle.  3) Deleted keys are available in ColumnFamilies - How to remove these keys.
  4) If we delete a record in clustered environment (In a single node), after that before sync with remaining nodes the node got failure means we wont have the data consistency - how to handle it. This database wont follow ACID properties, how to handle this one for a huge ERP kind of application.
 5) Suggest me some GUI clients for Cassandra Database - to cross check the records in backend.   /RegardsSekhar.  

Re: Schema Questions?

[Aaron Morton]

There is a system_update_column_family(CDef) function on the trunk, so it will be in 0.7-beta2. Take a look at the interface/cassandra.thrift .AOn 29 Sep, 2010,at 09:31 AM, Frank LoVecchio  wrote:I understand adding column family configuration fields using thrift in Pelops for system_add_column_family, and even renaming column families, but what about modifying the actual column family configuration values, e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)?
On Tue, Sep 28, 2010 at 1:17 PM, Peter Schuller  wrote:
> Question regarding schema set-up and modification:  I just set up the newest
> nightly build of 0.7, imported our .yaml file after starting Cassandra, and
> everything is honky-dorey.  We're not sure if Pelops is capable of modifying
> shema definitions, but it doesn't appear any changes we make to the .yaml
> file after the initial import go through (re-importing using schematool
> doesn't work).

Expected in 0.7 where the configuration file is essentially just an example:

   http://wiki.apache.org/cassandra/FAQ#no_keyspaces
   http://wiki.apache.org/cassandra/LiveSchemaUpdates

> How could I change, for example, the rows_cached or name
> fields below on the fly without losing data?  Is this possible?

Check out the system_* methods at the bottom of
interface/cassandra.thrift. I believe these are in working order for
0.7

--
/ Peter Schuller

Re: Schema Questions?

[Frank LoVecchio]

Ok,

I tried that method in Pelops, and this is the error I get when trying to to
update a Column Family Type from Standard to Super:

 InvalidRequestException(why:types do not match.)
   at
org.apache.cassandra.thrift.Cassandra$system_update_column_family_result.read(Cassandra.java:27741)
   at
org.apache.cassandra.thrift.Cassandra$Client.recv_system_update_column_family(Cassandra.java:1579)
   at
org.apache.cassandra.thrift.Cassandra$Client.system_update_column_family(Cassandra.java:1554)

I have last night's build of Cassandra.

On Tue, Sep 28, 2010 at 2:55 PM, Aaron Morton wrote:

> There is a system_update_column_family(CDef) function on the trunk, so it
> will be in 0.7-beta2. Take a look at the interface/cassandra.thrift .
>
>
> A
>
> On 29 Sep, 2010,at 09:31 AM, Frank LoVecchio  wrote:
>
> I understand adding column family configuration fields using thrift in
> Pelops for system_add_column_family, and even renaming column families, but
> what about modifying the actual column family configuration values,
> e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)?
>
>
>
> On Tue, Sep 28, 2010 at 1:17 PM, Peter Schuller <
> peter.schul...@infidyne.com> wrote:
>
>> > Question regarding schema set-up and modification:  I just set up the
>> newest
>> > nightly build of 0.7, imported our .yaml file after starting Cassandra,
>> and
>> > everything is honky-dorey.  We're not sure if Pelops is capable of
>> modifying
>> > shema definitions, but it doesn't appear any changes we make to the
>> .yaml
>> > file after the initial import go through (re-importing using schematool
>> > doesn't work).
>>
>> Expected in 0.7 where the configuration file is essentially just an
>> example:
>>
>>   http://wiki.apache.org/cassandra/FAQ#no_keyspaces
>>   http://wiki.apache.org/cassandra/LiveSchemaUpdates
>>
>> > How could I change, for example, the rows_cached or name
>> > fields below on the fly without losing data?  Is this possible?
>>
>> Check out the system_* methods at the bottom of
>> interface/cassandra.thrift. I believe these are in working order for
>> 0.7
>>
>> --
>> / Peter Schuller
>>
>
>

Re: Schema Questions?

[Aaron Morton]

That looks like an error thrown from CFMetaData, telling you it's not possible to change a Standard CF to a Super CF. Which seems logical to me, how would it move the data from a standard CF to a super CF?  You'll need to create a new Super CF and migrate the data manually. AaronOn 29 Sep, 2010,at 10:15 AM, Frank LoVecchio  wrote:Ok, I tried that method in Pelops, and this is the error I get when trying to to update a Column Family Type from Standard to Super: InvalidRequestException(why:types do not match.)
       at org.apache.cassandra.thrift.Cassandra$system_update_column_family_result.read(Cassandra.java:27741)       at org.apache.cassandra.thrift.Cassandra$Client.recv_system_update_column_family(Cassandra.java:1579)
       at org.apache.cassandra.thrift.Cassandra$Client.system_update_column_family(Cassandra.java:1554)I have last night's build of Cassandra.
On Tue, Sep 28, 2010 at 2:55 PM, Aaron Morton  wrote:
There is a system_update_column_family(CDef) function on the trunk, so it will be in 0.7-beta2. Take a look at the interface/cassandra.thrift .A
On 29 Sep, 2010,at 09:31 AM, Frank LoVecchio  wrote:I understand adding column family configuration fields using thrift in Pelops for system_add_column_family, and even renaming column families, but what about modifying the actual column family configuration values, e.g. read_repair_chance: 1.0 to read_repair_chance: .5 (as a poor example)?

On Tue, Sep 28, 2010 at 1:17 PM, Peter Schuller  wrote:

> Question regarding schema set-up and modification:  I just set up the newest
> nightly build of 0.7, imported our .yaml file after starting Cassandra, and
> everything is honky-dorey.  We're not sure if Pelops is capable of modifying
> shema definitions, but it doesn't appear any changes we make to the .yaml
> file after the initial import go through (re-importing using schematool
> doesn't work).

Expected in 0.7 where the configuration file is essentially just an example:

   http://wiki.apache.org/cassandra/FAQ#no_keyspaces
   http://wiki.apache.org/cassandra/LiveSchemaUpdates

> How could I change, for example, the rows_cached or name
> fields below on the fly without losing data?  Is this possible?

Check out the system_* methods at the bottom of
interface/cassandra.thrift. I believe these are in working order for
0.7

--
/ Peter Schuller

avro + cassandra + ruby

[Gabor Torok]

Hi,
I'm attempting to use avro to talk to cassandra because the ruby thrift 
client's read performance is pretty bad (I measured 4x slower than java).

However, I run into a problem when calling multiget_slice. 
The server gives a KeyspaceNotDefinedException because 
clientState.getKeyspace() returns null.
It seems this is because ClientState stores the keyspace in a ThreadLocal.

I call set_keyspace and clientState stores the keyspace value. I guess the next 
avro call to multiget_slice runs in a different thread so it can't retrieve the 
value.

In ruby, I use Avro::IPC::HTTPTransceiver as the transport which I believe is a 
stateless transport. I also tried SocketTransport, but that died with a malloc 
exception.

Is this a problem with the ruby avro library (I use avro 1.4.0), or how the 
server handles avro threads?
Any help would be appreciated!

Thanks,
--Gabor

Re: avro + cassandra + ruby

[Ryan King]

On Tue, Sep 28, 2010 at 4:06 PM, Gabor Torok
 wrote:
> Hi,
> I'm attempting to use avro to talk to cassandra because the ruby thrift 
> client's read performance is pretty bad (I measured 4x slower than java).

Only 4x feels like a win. :)

One thing you should try is to make thrift use
BinaryProtocolAccelerated, rather than the pure-ruby implementation
(we should change the default).

-ryan

> However, I run into a problem when calling multiget_slice.
> The server gives a KeyspaceNotDefinedException because 
> clientState.getKeyspace() returns null.
> It seems this is because ClientState stores the keyspace in a ThreadLocal.
>
> I call set_keyspace and clientState stores the keyspace value. I guess the 
> next avro call to multiget_slice runs in a different thread so it can't 
> retrieve the value.
>
> In ruby, I use Avro::IPC::HTTPTransceiver as the transport which I believe is 
> a stateless transport. I also tried SocketTransport, but that died with a 
> malloc exception.
>
> Is this a problem with the ruby avro library (I use avro 1.4.0), or how the 
> server handles avro threads?
> Any help would be appreciated!
>
> Thanks,
> --Gabor
>

Wed: Seattle Scalability / Hadoop / NoSQL Meetup: Killer Guests!

[Bradford Stephens]

Hey guys!

Just wanted to let you know that Wednesday's meetup is going to have
some *fantastic* speakers from around the world :) You need to come!

http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/calendar/13704368/
Wednesday, 7pm, Amazon SLU.

First we have Tim Anglade, who flew here all the way from Paris,
France. He's one of the foremost advocates of NoSQL, and the founder
of "A NoSQL Summer". His talks are always thought-provoking, lovely to
look at, and informative. He'll be talking about the lessons learned
and  mistakes of Hadoop and NoSQL, and what they'll look like in the
future.

We'll also have Ben Hendrickson from SEOMoz, who will talk about their
home-built, fascinating distributed DB.  IF you've ever wanted to know
what it takes to build a database from scratch, you'll find this to be
quite enlightening.

And as always, there's delicious German beer and bratwurst at
Fiererabend, 422 Yale Ave N., afterward.

http://www.meetup.com/Seattle-Hadoop-HBase-NoSQL-Meetup/calendar/13704368/

Cheers,
Bradford

-- 
Bradford Stephens,
Founder, Drawn to Scale
drawntoscalehq.com
727.697.7528

http://www.drawntoscalehq.com --  The intuitive, cloud-scale data
solution. Process, store, query, search, and serve all your data.

http://www.roadtofailure.com -- The Fringes of Scalability, Social
Media, and Computer Science

Re: Is the secondary index maintained synchronously in 0.7

[Alvin UW]

Why the secondary index is not updated absolutely atomically?

2010/9/2 Jonathan Ellis 

> yes, it is updated atomically (but not in isolation, it's possible for
> a client to see changes to one but not the other temporarily)
>
> On Thu, Sep 2, 2010 at 1:47 PM, Alvin Jin  wrote:
> >
> > Hello,
> >
> > I was thinking the details of the secondary index in 0.7.
> > Will it be updated atomically with its base table?
> >
> > Any explaination the on secondary index is appreciated.
> > Thanks.
> >
> > --
> > View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Is-the-secondary-index-maintained-synchronously-in-0-7-tp5492798p5492798.html
> > Sent from the cassandra-u...@incubator.apache.org mailing list archive
> at Nabble.com.
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Re: Is the secondary index maintained synchronously in 0.7

[Jonathan Ellis]

because it's not worth the penalty to concurrency

On Tue, Sep 28, 2010 at 9:02 PM, Alvin UW  wrote:
> Why the secondary index is not updated absolutely atomically?
>
> 2010/9/2 Jonathan Ellis 
>>
>> yes, it is updated atomically (but not in isolation, it's possible for
>> a client to see changes to one but not the other temporarily)
>>
>> On Thu, Sep 2, 2010 at 1:47 PM, Alvin Jin  wrote:
>> >
>> > Hello,
>> >
>> > I was thinking the details of the secondary index in 0.7.
>> > Will it be updated atomically with its base table?
>> >
>> > Any explaination the on secondary index is appreciated.
>> > Thanks.
>> >
>> > --
>> > View this message in context:
>> > http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Is-the-secondary-index-maintained-synchronously-in-0-7-tp5492798p5492798.html
>> > Sent from the cassandra-u...@incubator.apache.org mailing list archive
>> > at Nabble.com.
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Couple of minor logging / error message things

[Aaron Morton]

Noticed these when working against the current 0.7.0 beta2 (#3) build...When sending a system_add_keyspace request with an invalid keyspace the response to the client is fine... (python)InvalidRequestException: InvalidRequestException(why='Invalid keyspace name: Test Keyspace 1285729085.78')However it's not logged in the system.log. Looking through the code it appears that none of the Config errors are logged at any level. It may be handy to log these as a record of attempted changes on the system.In thrift/CassandraServer system_add_keyspace() the strategy_class string from the KsDef is used to load a class. The ClassNotFoundError is then caught and used to build an InvalidRequestException. If the strategy_class is missing or empty, the error returned to the client is (python)InvalidRequestException: InvalidRequestException(why='')Aaron

Re: avro + cassandra + ruby

[Gabor Torok]

Thanks, that made things better by about 30%. Unfortunately for me that's still 
unacceptable... :-(

I feel like I'm doing something wrong with avro (see my original post). Was 
anyone able to make it work?

Re: Couple of minor logging / error message things

[Jonathan Ellis]

On Tue, Sep 28, 2010 at 10:28 PM, Aaron Morton  wrote:
> Noticed these when working against the current 0.7.0 beta2 (#3) build...
> When sending a system_add_keyspace request with an invalid keyspace the
> response to the client is fine...
>
> (python)
> InvalidRequestException: InvalidRequestException(why='Invalid keyspace name:
> Test Keyspace 1285729085.78')
> However it's not logged in the system.log.

We don't log any InvalidRequest.  Not sure if it's worth
special-casing system_ methods there.

> In thrift/CassandraServer system_add_keyspace() the strategy_class string
> from the KsDef is used to load a class. The ClassNotFoundError is then
> caught and used to build an InvalidRequestException. If the strategy_class
> is missing or empty, the error returned to the client is
> (python)
> InvalidRequestException: InvalidRequestException(why='')

Can you create a ticket for this?

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Truncate + Snapshot + Cannot Allocate Memory == Timeout

[Aaron Morton]

Running on the current 0.7.0 beta2 #3. On my dev workstation (unubtu 10.4) I accidentally let cassandra have 4GB and free memory got down to <100 MB . I was running some tests that added a few (< 100) rows to the CF's in Keyspace1, then Truncated the CF's for the next tests.  I got the following error in the cassandra system.log DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,166 CassandraServer.java (line 954) truncating Standard1 in Keyspace1DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,166 StorageProxy.java (line 759) Starting a blocking truncate operation on keyspace Keyspace1, CF DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,166 StorageProxy.java (line 774) Starting to send truncate messages to hosts [localhost/127.0.0.1]DEBUG [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 TruncateVerbHandler.java (line 50) Applying Truncation(keyspace='Keyspace1', cf='Standard1')DEBUG [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 Tablejava (line 630) Truncating... INFO [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 ColumnFamilyStore.java (line 459) switching in a fresh Memtable for Standard1 at CommitLogContext(file='/local1/junkbox/cassandra/commitlog/CommitLog-1285728080403.log', position=10022050) INFO [MUTATION_STAGE:25] 2010-09-29 16:44:39,166 ColumnFamilyStore.java (line 771) Enqueuing flush of memtable-standa...@1701009673(40104 bytes, 1152 operations)DEBUG [pool-1-thread-44] 2010-09-29 16:44:39,167 StorageProxy.java (line 780) Sent all truncate messages, now waiting for 1 responses INFO [FLUSH-WRITER-POOL:1] 2010-09-29 16:44:39,167 Memtable.java (line 150) Writing memtable-standa...@1701009673(40104 bytes, 1152 operations) INFO [FLUSH-WRITER-POOL:1] 2010-09-29 16:44:39,268 Memtable.java (line 157) Completed flushing /local1/junkbox/cassandra/data/Keyspace1/Standard1-e-10-Data.dbDEBUG [CompactionExecutor:1] 2010-09-29 16:44:39,268 CompactionManager.java (line 95) Checking to see if compaction of Standard1 would be usefulDEBUG [COMMIT-LOG-WRITER] 2010-09-29 16:44:39,268 CommitLog.java (line 422) discard completed log segments for CommitLogContext(file='/local1/junkbox/cassandra/commitlog/CommitLog-1285728080403.log', position=10022050), column family 1000.DEBUG [COMMIT-LOG-WRITER] 2010-09-29 16:44:39,268 CommitLog.java (line 448) Marking replay position 10022050 on commit log CommitLogSegment(/local1/junkbox/cassandra/commitlog/CommitLog-1285728080403.log)ERROR [MUTATION_STAGE:25] 2010-09-29 16:44:39,341 AbstractCassandraDaemon.java (line 88) Fatal exception in thread Thread[MUTATION_STAGE:25,5,main]java.io.IOError: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory        at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1368)        at org.apache.cassandra.db.ColumnFamilyStore.truncate(ColumnFamilyStore.java:1511)        at org.apache.cassandra.db.Table.truncate(Table.java:633)        at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:54)        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50)        at javautil.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)        at java.lang.Thread.run(Thread.java:619)Caused by: java.io.IOException: Cannot run program "ln": java.io.IOException: error=12, Cannot allocate memory        at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)        at org.apache.cassandra.io.util.FileUtils.createHardLinkWithExec(FileUtils.java:263)        at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:229)        at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1360)        ... 7 moreCaused by: java.io.IOException: java.io.IOException: error=12, Cannot allocate memory        at java.lang.UNIXProcess.(UNIXProcess.java:148)        at java.lang.ProcessImpl.start(ProcessImpl.java:65)        at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)        ... 10 moreDEBUG [GC inspection] 2010-09-29 16:44:39,539 GCInspector.java (line 131) GC for ParNew: 43 ms, 29106728 reclaimed leaving 3567200784 used; max is 5498404864DEBUG [pool-1-thread-44] 2010-09-29 16:44:49,189 ClientState.java (line 107) logged out: #Fair enough, though it appeared as a timeout on the client (with a 10 second timeout)   File "/tech/home//git_home/trojan/trojan/cassandra/Cassandrapy", line 846, in truncate    self.recv_truncate()  File "/tech/home//git_home/trojan/trojan/cassandra/Cassandra.py", line 857, in recv_truncate    (fname, mtype, rseqid) = self._iprot.readMessageBegin()  File "/tech/home//git_home/trojan/trojan/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin    sz = self.readI32()    chunk = self.read(sz-have)  File "/tech/home//git_home/trojan/trojan/thrift/transport/TSocket.py", line 92, in read    buff = self.handle.recv(sz)timeout: timed outHere's the output of ulimit -a core file size          (bloc

Re: Truncate + Snapshot + Cannot Allocate Memory == Timeout

[Jonathan Ellis]

On Tue, Sep 28, 2010 at 11:25 PM, Aaron Morton  wrote:
> 1) Is the memory error just a result of me letting my machine run stupidly
> low on memory?

No, it's the JVM forking to run ln.  Enable overcommit, or get JNA so
it does the link w/ native code instead.

> 2) Should it have returned an ApplicationError or some such in this case?
> The code in ColumnFamilyStore:1368 is catching the IOException from the call
> to FileUtils.createHardLink and wrapping it in an IOError. However the code
> in TruncateVerbHandler:56 is looking for the IOException.

That does sound like a bug.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: Is there a debian 0.6.1 install package archived anywhere?

[Kyusik Chung]

Thanks Eric.  Im not sure why I couldnt locate it earlierthought I had 
looked in all the directories.

Kyusik Chung