Re: SV: Help with getting Key range with some column limitations

[Jone Lura]

 Thanks for you suggestions.

I tried to iterate them, however I could not get it to work (pretty sure 
its my code). Im still not to familiar with Cassandra, so could you 
provide a small example?


The key count could be up to atleast 20k and maybe more, and users 
should not wait for more than 10 seconds for their map, so I also want 
to investigate the plugin suggestion. Does the plugin exist? or do I 
have to develop it myself? Are there any documentation on plugin 
development for Cassandra?


Best regards

Jone


On 19/08/2010 08:42, Thorvaldsson Justus wrote:


You should iterate through them, get 200 then go get the next 200 and 
so on.


Also if checking a bounding box to another.. perhaps try sorting them 
so you could start looking at both ends, perhaps make the iteration 
smaller until match somehow?


Just my simple coins, also upgrading will probably be needed to 
iterate through RP because of bugs. But that should be simple enough 
to 6.4


/Justus

*Från:* Jone Lura [mailto:jone.l...@ecc.no]
*Skickat:* den 18 augusti 2010 20:32
*Till:* user@cassandra.apache.org
*Ämne:* Help with getting Key range with some column limitations

Hi,

We are trying to implement Cassandra to replace one of our biggest SQL 
tables, and so far we got it working.


However, for testing I'm using Cassandra 0.6.2, Java and Pelops. 
(Pelops not that important for my question) and need suggestions on 
how to solve a problem retrieving a key range based on the following.






 ...



The super column got columns for longitude and latitude.

 1. Need to get get max long number for key

 2. The key should also have supercolumns latitude and longitude 
column intersecting (inside) with a given boundingbox.


Currently I'm doing like this

KeyRange keyRange = new KeyRange();

keyRange.setStart_key("");

keyRange.setEnd_key("");

keyRange.setCount(700);

And checking every row in db if it match my bounding box.

But there are a lot more than 700 keys.. and if i set a higher count, 
the get_range_slice get a Timeout Exception.


Any ideas?

Best Regards

Jone

Re: Node OOM Problems

[Wayne]

The NullPointerException does not crash the node. It only makes it flap/go
down a for short period and then it comes back up. I do not see anything
abnormal in the system log, only that single error in the cassandra.log.


On Thu, Aug 19, 2010 at 11:42 PM, Peter Schuller <
peter.schul...@infidyne.com> wrote:

> > What is my "live set"?
>
> Sorry; that meant the "set of data acually live (i.e., not garbage) in
> the heap". In other words, the amount of memory truly "used".
>
> > Is the system CPU bound given the few statements
> > below? This is from running 4 concurrent processes against the node...do
> I
> > need to throttle back the concurrent read/writers?
> >
> > I do all reads/writes as Quorum. (Replication factor of 3).
>
> With quorom and 0.6.4 I don't think unthrottled writes are expected to
> cause a problem.
>
> > The memtable threshold is the default of 256.
> >
> > All caching is turned off.
> >
> > The database is pretty small, maybe a few million keys (2-3) in 4 CFs.
> The
> > key size is pretty small. Some of the rows are pretty fat though (fatter
> > than I thought). I am saving secondary indexes in separate CFs and those
> are
> > the large rows that I think might be part of the problem. I will restart
> > testing turning these off and see if I see any difference.
> >
> > Would an extra fat row explain repeated OOM crashes in a row? I have
> finally
> > got the system to stabilize relatively and I even ran compaction on the
> bad
> > node without a problem (still no row size stats).
>
> Based on what you've said so far, the large rows are the only thing I
> would suspect may be the cause. With the amount of data and keys you
> say you have, you should definitely not be having memory issues with
> an 8 gig heap as a direct result of the data size/key count. A few
> million keys is not a lot at all; I still claim you should be able to
> handle hundreds of millions at least, from the perspective of bloom
> filters and such.
>
> So your plan to try it without these large rows is probably a good
> idea unless some else has a better idea.
>
> You may want to consider trying 0.7 betas too since it has removed the
> limitation with respect to large rows, assuming you do in fact want
> these large rows (see the CassandraLimitations wiki page that was
> posted earlier in this thread).
>
> > I now have several other nodes flapping with the following single error
> in
> > the cassandra.log
> > Error: Exception thrown by the agent : java.lang.NullPointerException
> >
> > I assume this is an unrelated problem?
>
> Do you have a full stack trace?
>
> --
> / Peter Schuller
>

SV: Replication factor and other schema changes in >= 0.7

[Thorvaldsson Justus]

KsDef
CfDef   <-has metadata
And perhaps ColumnDef

how to make a ksdef---
KsDef k = new KsDef();
k.setName(keyspacename);
k.setReplication_factor(replicafactor);
k.setStrategy_class("org.apache.cassandra.locator.RackUnawareStrategy");
List cfDefs = new ArrayList();
k.setCf_defs(cfDefs);
c.system_add_keyspace(k);

/Justus
www.justus.st

Från: Andres March [mailto:ama...@qualcomm.com]
Skickat: den 20 augusti 2010 01:01
Till: user@cassandra.apache.org
Ämne: Replication factor and other schema changes in >= 0.7

How should we go about changing the replication factor and other keyspace 
settings now that it and other KSMetaData are no longer managed in 
cassandra.yaml?

I found makeDefinitionMutation() in the Migration class and see that it is 
called for the other schema migrations.  There just seems to be a big gap in 
the management API for the KSMetaData we might want to change.
--
Andres March
ama...@qualcomm.com
Qualcomm Internet Services

Re: Data Corruption

[Waqas Badar]

In fact, on one node hard disk filled up so thats why we have to shift
cassandra manually on another machine. Can you please tell any work
around to restore data?

On Thu, 2010-08-19 at 09:56 -0500, Jonathan Ellis wrote:

> You're moving data around manually?  That sounds like a good way to
> confuse Cassandra's replication.
> 
> On Thu, Aug 19, 2010 at 4:33 AM, Waqas Badar
>  wrote:
> > We are observing a strange behavior of Cassandra. We have a ring of two
> > nodes. When we inserts data in cassandra then old data after some entries
> > get vanished. Please note that it is not a data loss, as when we move that
> > data to separate node then all data is shown.  We are using Cassandra 0.6.3
> > and Ubuntu Server 8.10
> 
> 
> 

Re: test

[mallikarj...@iss-global.com]

Don't keep these test mails and don't waist the time of others.



ChingShen wrote:
test 

Re: questions regarding read and write in cassandra

[Benjamin Black]

More recent.  Newest timestamp always wins.  And I am moving this to
the user list (again) so it can be with all its friendly threads on
the exact same topic.

On Thu, Aug 19, 2010 at 10:22 AM, Maifi Khan  wrote:
> Hi David
> Thanks for your reply.
> But what happens if I read and get 2 nodes has value 10 with older
> time stamp and the third node has 20 with more recent time stamp?
> Will cassandra return 10(majority) or 20(more recent)?
>
> thanks
> Maifi
>
> On Thu, Aug 19, 2010 at 1:20 PM, David Timothy Strauss
>  wrote:
>> The quorum write would fail, but the data would not be rolled back. Assuming 
>> the offline nodes recover, the data would eventually replicate.
>>
>> This question belongs on the user list, though.
>>
>> -Original Message-
>> From: Maifi Khan 
>> Date: Thu, 19 Aug 2010 13:00:47
>> To: 
>> Reply-To: d...@cassandra.apache.org
>> Subject: questions regarding read and write in cassandra
>>
>> Hi
>> I have a question in the following scenario.
>> Say we have 10 nodes, Replication factor is 5.
>> Now, say, for Row X, Column Y, data is replicated to node 1,2,3,4,5
>> and current value is 10
>> Say, I issue a write command with value “20” to Row X, column Y with
>> quorum(n/2+1=3 nodes).  Say it updated 1 and 2 and failed to update
>> any other node. So it failed to write to 3 nodes. What happens in such
>> scenario?
>>
>> Q: Will the user returned failed?
>>
>> Now, assuming that the write failed.
>> What value will I see if I want to read the same cell with Quorum?
>> Now, say I read the data with quorum. It read from 1, 4, 5 and see
>> that node 1 has the most recent data (“20” which is still there as
>> cassandra does not roll back).
>> Will it will return the data “20” to the user or will it return the
>> earlier value 10 as it is returned by the node 4 and 5?
>>
>> Also, does read repair tries to propagate 20 to all the replicas
>> although cassandra returned failed to the user?
>>
>>
>> thanks
>>
>

Re: Job Opportunity in Europe (Nosql, hadoop, crawling)

[sharanabasava raddi]

Hi,
I have worked on Cassandra, Thrift and Hector, and I did it for 3 months, I
have written code for loading and fetching data from Cassandra in single
node, I want to work on this domain. Will u provide me the environment to
work on this?

On Wed, Aug 18, 2010 at 8:29 PM, Thibaut Britz  wrote:

> Hi,
>
> We are searching at least 3 more developers in the fields of search &
> automatic content/site extraction, crawling, duplicate content, news/spam
> detection.
>
> We do content fetching and aggregation (news,message boards, blogs, ...)
> for market research institutes, media analytics companies, etc...
>
> We are still relative small (mostly Harvard, ETH Zurich, and TU Munich
> graduates), so you are still able to actively shape our company.
>
> You will work on interesting problems/features/products involving large
> datasets and write production code for large scale application tools like
> apache hadoop, cassandra, hbase, zookeeper running on over 50 servers, and
> processing terabytes of data each day.
>
>
> If you are from abroad, and want to experience a different culture for a
> few months/years (some even stayed here their entire life ;)) in a small
> french/german/english speaking country, why not join?
>
> We can also offer internships for a few months (probably 6).
>
> There are more details at our blog, or just contact me for further details
> under my email thibaut.br...@trendiction.com:
>
> http://blog.trendiction.com/tag/jobs
>
> Thanks,
> Thibaut
>

test

[ChingShen]

test

Re: Cassandra w/ Hadoop

[Mark]

 On 8/19/10 11:14 AM, Mark wrote:

 On 8/19/10 10:23 AM, Jeremy Hanna wrote:
I would check out http://wiki.apache.org/cassandra/HadoopSupport for 
more info.  I'll try to explain a bit more here, but I don't think 
there's a tutorial out there yet.


For input:
- configure your main class where you're starting the mapreduce job 
the way the word_count is configured (with either storage-conf or in 
your code via the ConfigHelper).  It will complain specifically about 
stuff you hadn't configured - esp. important is your cassandra server 
and port.
- the inputs to your mapper are going to be what's coming from 
cassandra - so your key with a map of row values
- you need to set your column name in your overridden setup method in 
your mapper
- for the reducer, nothing really changes from a normal map/reduce, 
unless you want to output to cassandra
- generally cassandra just provides an inputformat and split classes 
to read from cassandra - you can find the guts in the 
org.apache.cassandra.hadoop package


For output:
- in your reducer, you could just write to cassandra directly via 
thrift.  there is a built-in outputformat coming in 0.7 but it still 
might change before 0.7 final - that will queue up changes so it will 
write large blocks all at once.



On Aug 19, 2010, at 12:07 PM, Mark wrote:

Are there any examples/tutorials on the web for reading/writing from 
Cassandra into/from Hadoop?


I found the example in contrib/word_count but I really can't make 
sense of it... a tutorial/explanation would help.

Thanks!
How does batching across all rows work? Does it just take an arbitrary 
start w/ a limit of x and then use the last key from that result as the 
next start? Does this work with RandomPartitioner?

Replication factor and other schema changes in >= 0.7

[Andres March]

 How should we go about changing the replication factor and other 
keyspace settings now that it and other KSMetaData are no longer managed 
in cassandra.yaml?


I found makeDefinitionMutation() in the Migration class and see that it 
is called for the other schema migrations.  There just seems to be a big 
gap in the management API for the KSMetaData we might want to change.

--
*Andres March*
ama...@qualcomm.com 
Qualcomm Internet Services

Re: Errors with Cassandra 0.7

[Alaa Zubaidi]

That was it..
Thanks Peter

Peter Schuller wrote:

I am trying to run Cassandra 0.7 and I am getting different errors: First it
was while calling client.insert and now while calling set_keyspace (see
below).



Are you perhaps not using a framed transport with thrift? Cassandra
0.7 uses framed by default; 0.6 did not.

  


--
Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 700
San Jose, CA 95110  USA
Tel: 408-283-5639 (or 408-280-7900 x5639)
fax: 408-938-6479
email: alaa.zuba...@pdf.com 

recovering a failed node does not seem to recover replicas

[Scott Dworkis]

following the failure handling process described here:

http://wiki.apache.org/cassandra/Operations

i don't at the end seem to have all the data... i have half as much being 
reported by nodetool ring as i started with.  i'm guessing the replicas 
are not being recovered.  however if i take the extra steps of repair and 
cleanup (as described by the alternative approach), i seem to get the same 
amount of data that i started with.  btw, then i do a gc to clean out the 
2.5x amount of disk usage consumed by repair and cleanup... this gc might 
be automatic with disks that are closer to full than mine.


so, i think the wiki isn't quite accurate, and probably should mention the 
need for at least 2.5x of what is reported by nodetool ring worth of 
storage to handle recovery of a failed node.


-scott

Re: Node OOM Problems

[Peter Schuller]

> Sorry; that meant the "set of data acually live (i.e., not garbage) in
> the heap". In other words, the amount of memory truly "used".

And to clarify further this is not the same as the 'used' reported by
GC statistics, except as printed after a CMS concurrent mark/sweep has
completed (and even then the number is not exact due to changes made
during the concurrent mark/sweep phase).

That's why I said "live" (a term often used in the context of GC; an
object which is "live" is one which is reachable by the application
and thus by definition not eligible for garbage collection).

-- 
/ Peter Schuller

Re: Node OOM Problems

[Peter Schuller]

> What is my "live set"?

Sorry; that meant the "set of data acually live (i.e., not garbage) in
the heap". In other words, the amount of memory truly "used".

> Is the system CPU bound given the few statements
> below? This is from running 4 concurrent processes against the node...do I
> need to throttle back the concurrent read/writers?
>
> I do all reads/writes as Quorum. (Replication factor of 3).

With quorom and 0.6.4 I don't think unthrottled writes are expected to
cause a problem.

> The memtable threshold is the default of 256.
>
> All caching is turned off.
>
> The database is pretty small, maybe a few million keys (2-3) in 4 CFs. The
> key size is pretty small. Some of the rows are pretty fat though (fatter
> than I thought). I am saving secondary indexes in separate CFs and those are
> the large rows that I think might be part of the problem. I will restart
> testing turning these off and see if I see any difference.
>
> Would an extra fat row explain repeated OOM crashes in a row? I have finally
> got the system to stabilize relatively and I even ran compaction on the bad
> node without a problem (still no row size stats).

Based on what you've said so far, the large rows are the only thing I
would suspect may be the cause. With the amount of data and keys you
say you have, you should definitely not be having memory issues with
an 8 gig heap as a direct result of the data size/key count. A few
million keys is not a lot at all; I still claim you should be able to
handle hundreds of millions at least, from the perspective of bloom
filters and such.

So your plan to try it without these large rows is probably a good
idea unless some else has a better idea.

You may want to consider trying 0.7 betas too since it has removed the
limitation with respect to large rows, assuming you do in fact want
these large rows (see the CassandraLimitations wiki page that was
posted earlier in this thread).

> I now have several other nodes flapping with the following single error in
> the cassandra.log
> Error: Exception thrown by the agent : java.lang.NullPointerException
>
> I assume this is an unrelated problem?

Do you have a full stack trace?

-- 
/ Peter Schuller

Re: Node OOM Problems

[Edward Capriolo]

On Thu, Aug 19, 2010 at 4:49 PM, Wayne  wrote:
> What is my "live set"? Is the system CPU bound given the few statements
> below? This is from running 4 concurrent processes against the node...do I
> need to throttle back the concurrent read/writers?
>
> I do all reads/writes as Quorum. (Replication factor of 3).
>
> The memtable threshold is the default of 256.
>
> All caching is turned off.
>
> The database is pretty small, maybe a few million keys (2-3) in 4 CFs. The
> key size is pretty small. Some of the rows are pretty fat though (fatter
> than I thought). I am saving secondary indexes in separate CFs and those are
> the large rows that I think might be part of the problem. I will restart
> testing turning these off and see if I see any difference.
>
> Would an extra fat row explain repeated OOM crashes in a row? I have finally
> got the system to stabilize relatively and I even ran compaction on the bad
> node without a problem (still no row size stats).
>
> I now have several other nodes flapping with the following single error in
> the cassandra.log
> Error: Exception thrown by the agent : java.lang.NullPointerException
>
> I assume this is an unrelated problem?
>
> Thanks for all of your help!
>
> On Thu, Aug 19, 2010 at 10:26 PM, Peter Schuller
>  wrote:
>>
>> So, these:
>>
>> >  INFO [GC inspection] 2010-08-19 16:34:46,656 GCInspector.java (line
>> > 116) GC
>> > for ConcurrentMarkSweep: 41615 ms, 192522712 reclaimed leaving
>> > 8326856720
>> > used; max is 8700035072
>> [snip]
>> > INFO [GC inspection] 2010-08-19 16:36:00,786 GCInspector.java (line 116)
>> > GC for ConcurrentMarkSweep: 37122 ms, 157488
>> > reclaimed leaving 8342836376 used; max is 8700035072
>>
>> ...show that you're live set is indeed very close to heap maximum, and
>> so concurrent mark/sweep phases run often freeing very little memory.
>> In addition the fact that it seems to take 35-45 seconds to do the
>> concurrent mark/sweep on an 8 gig heap on a modern system suggests
>> that you are probably CPU bound in cassandra at the time (meaning GC
>> is slower).
>>
>> In short you're using too much memory in comparison to the maximum
>> heap size. The expected result is to either get an OOM, or just become
>> too slow due to excessive GC activity (usually the latter followed by
>> the former).
>>
>> Now, the question is what memory is used *for*, and why. First off, to
>> get that out of the way, are you inserting with consistency level
>> ZERO? I am not sure whether it applies to 0.6.4 or not but there used
>> to be an issue involving writes at consistency level ZERO not being
>> throttled at all, meaning that if you threw writes at the system
>> faster than it would handle them, you would accumulate memory use. I
>> don't believe this is a problem with CL.ONE and above, even in 0.6.4
>> (but someone correct me if I'm wrong).
>>
>> (As an aside: I'm not sure whether the behavior was such that it might
>> explain OOM on restart as a result of accumulated commitlogs that get
>> replayed faster than memtable flushing happens. Perhaps not, not
>> sure.)
>>
>> In any case, the most important factors are what you're actually doing
>> with the cluster, but you don't say much about the data. In particular
>> how many rows and colums you're populating it with.
>>
>> The primary users of large amounts of memory in cassandra include
>> (hopefully I'm not missing something major);
>>
>> * bloom filters that are used to efficiently avoid doing I/O on
>> sstables that do not contain relevant data. the size of bloom filters
>> scale linearly with the number of row keys (not columns right? I don't
>> remember). so here we have an expected permanent, but low, memory use
>> as a result of a large database. how large is your database? 100
>> million keys? 1 billion? 10 billion?
>>
>> * the memtables; the currently active memtable and any memtables
>> currently undergoing flushing. the size of these are directly
>> controllable in the configuration file. make sure they are reasonable.
>> (If you're not sure at all, with an 8 gig heap I'd say <= 512 mb is a
>> reasonable recommendation unless you have a reason to make them
>> larger)
>>
>> * row cache and key cache, both controllable in the configuration. in
>> particular the row cache can be huge if you have configured it as
>> such.
>>
>> * to some extent unflushed commitlogs; the commit log rotation
>> threshold controls this. the default value is low enough that it
>> should not be your culprit
>>
>> So the question is what you're usage is like. How many unique rows do
>> you have? How many columns? The data size in and of itself should not
>> matter much to memory use, except of course that extremely large
>> individual values will be relevant to transient high memory use when
>> they are read/written.
>>
>> In general, lacking large row caches and such things, you should be
>> able to have hundreds of millions of entries on an 8 gb heap, assuming
>> reasonably sized keys.
>>
>> --
>> / Pet

Re: Errors with Cassandra 0.7

[Peter Schuller]

> I am trying to run Cassandra 0.7 and I am getting different errors: First it
> was while calling client.insert and now while calling set_keyspace (see
> below).

Are you perhaps not using a framed transport with thrift? Cassandra
0.7 uses framed by default; 0.6 did not.

-- 
/ Peter Schuller

Errors with Cassandra 0.7

[Alaa Zubaidi]

Hi,

I am trying to run Cassandra 0.7 and I am getting different errors: 
First it was while calling client.insert and now while calling 
set_keyspace (see below).

Any help is appreciated.

Exception in thread "main" org.apache.thrift.transport.TTransportException
   at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)

   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
   at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:369)
   at 
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:295)
   at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:202)
   at 
org.apache.cassandra.thrift.Cassandra$Client.recv_set_keyspace(Cassandra.java:439)
   at 
org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.java:424)
   at 
rtsim.CassandraInterface.ConnectToKeyspace(CassandraInterface.java:216)

   at rtsim.Main.main(Main.java:308)


Note: I get the following when I start Cassandra:
*10/08/19 12:58:26 INFO utils.Mx4jTool: Will not load MX4J, 
mx4j-tools.jar is not in the classpath*




--
Alaa Zubaidi

Re: Node OOM Problems

[Wayne]

What is my "live set"? Is the system CPU bound given the few statements
below? This is from running 4 concurrent processes against the node...do I
need to throttle back the concurrent read/writers?

I do all reads/writes as Quorum. (Replication factor of 3).

The memtable threshold is the default of 256.

All caching is turned off.

The database is pretty small, maybe a few million keys (2-3) in 4 CFs. The
key size is pretty small. Some of the rows are pretty fat though (fatter
than I thought). I am saving secondary indexes in separate CFs and those are
the large rows that I think might be part of the problem. I will restart
testing turning these off and see if I see any difference.

Would an extra fat row explain repeated OOM crashes in a row? I have finally
got the system to stabilize relatively and I even ran compaction on the bad
node without a problem (still no row size stats).

I now have several other nodes flapping with the following single error in
the cassandra.log
Error: Exception thrown by the agent : java.lang.NullPointerException

I assume this is an unrelated problem?

Thanks for all of your help!

On Thu, Aug 19, 2010 at 10:26 PM, Peter Schuller <
peter.schul...@infidyne.com> wrote:

> So, these:
>
> >  INFO [GC inspection] 2010-08-19 16:34:46,656 GCInspector.java (line 116)
> GC
> > for ConcurrentMarkSweep: 41615 ms, 192522712 reclaimed leaving 8326856720
> > used; max is 8700035072
> [snip]
> > INFO [GC inspection] 2010-08-19 16:36:00,786 GCInspector.java (line 116)
> GC for ConcurrentMarkSweep: 37122 ms, 157488
> > reclaimed leaving 8342836376 used; max is 8700035072
>
> ...show that you're live set is indeed very close to heap maximum, and
> so concurrent mark/sweep phases run often freeing very little memory.
> In addition the fact that it seems to take 35-45 seconds to do the
> concurrent mark/sweep on an 8 gig heap on a modern system suggests
> that you are probably CPU bound in cassandra at the time (meaning GC
> is slower).
>
> In short you're using too much memory in comparison to the maximum
> heap size. The expected result is to either get an OOM, or just become
> too slow due to excessive GC activity (usually the latter followed by
> the former).
>
> Now, the question is what memory is used *for*, and why. First off, to
> get that out of the way, are you inserting with consistency level
> ZERO? I am not sure whether it applies to 0.6.4 or not but there used
> to be an issue involving writes at consistency level ZERO not being
> throttled at all, meaning that if you threw writes at the system
> faster than it would handle them, you would accumulate memory use. I
> don't believe this is a problem with CL.ONE and above, even in 0.6.4
> (but someone correct me if I'm wrong).
>
> (As an aside: I'm not sure whether the behavior was such that it might
> explain OOM on restart as a result of accumulated commitlogs that get
> replayed faster than memtable flushing happens. Perhaps not, not
> sure.)
>
> In any case, the most important factors are what you're actually doing
> with the cluster, but you don't say much about the data. In particular
> how many rows and colums you're populating it with.
>
> The primary users of large amounts of memory in cassandra include
> (hopefully I'm not missing something major);
>
> * bloom filters that are used to efficiently avoid doing I/O on
> sstables that do not contain relevant data. the size of bloom filters
> scale linearly with the number of row keys (not columns right? I don't
> remember). so here we have an expected permanent, but low, memory use
> as a result of a large database. how large is your database? 100
> million keys? 1 billion? 10 billion?
>
> * the memtables; the currently active memtable and any memtables
> currently undergoing flushing. the size of these are directly
> controllable in the configuration file. make sure they are reasonable.
> (If you're not sure at all, with an 8 gig heap I'd say <= 512 mb is a
> reasonable recommendation unless you have a reason to make them
> larger)
>
> * row cache and key cache, both controllable in the configuration. in
> particular the row cache can be huge if you have configured it as
> such.
>
> * to some extent unflushed commitlogs; the commit log rotation
> threshold controls this. the default value is low enough that it
> should not be your culprit
>
> So the question is what you're usage is like. How many unique rows do
> you have? How many columns? The data size in and of itself should not
> matter much to memory use, except of course that extremely large
> individual values will be relevant to transient high memory use when
> they are read/written.
>
> In general, lacking large row caches and such things, you should be
> able to have hundreds of millions of entries on an 8 gb heap, assuming
> reasonably sized keys.
>
> --
> / Peter Schuller
>

Re: Node OOM Problems

[Peter Schuller]

> of a rogue large row is one I never considered. The largest row on the other
> nodes is as much as 800megs. I can not get a cfstats reading on the bad node

WIth 0.6 I can definitely see this being a problem if I understand its
behavior correctly (I have not actually used 0.6 even for testing). In
particular such amounts of data is likely to end up directly in the
old generation in the GC (normally the young generation is smaller
than 800 mb, and that does not take into account the time it takes to
actually read and process those large rows and the likelyhood of a
young-generation gc triggering anyway due to other normal activity).
Having a single value be 10% of the total heap size is likely to be
problematic in general (that could be said in some cases (e.g. 32 bit
virtual memory space and fragmentation issues) for e.g.
malloc()/free() too; algorithms solving general allocation problems
are often not very good at dealing with extreme outliers).

> so do not know how big its largest row is. I will raise memory to 16gb and
> see if that makes a difference. I had though that the java heap sizes that
> high had issues on their own in term of GC.

The garbage collector may or may not have issues in particular cases,
and to some extent the heap size is definitely a factor. However, a
lot of other things play in, including the application's overall
allocation behavior and pointer writing behavior. A large heap size in
and of itself should not be a huge problem; if you combine a very
large heap size with lots of allocation and lots of behavior that is
difficult for the particular GC to deal with, you may be more likely
to have problems.

My gut feeling with Cassandra is that I expect it to be fine, with the
worst case being having to tweak GC settings to e.g. make the
concurrent mark/sweep phases kick in earlier. In other words I would
not expect Cassandra to be an application where it becomes problematic
to keep CMS pause times down. However, I have no hard evidence of
that.

I'd be very interested to hear if people have other experiences in
production environments with very large heap sizes.

-- 
/ Peter Schuller

Re: Node OOM Problems

[Peter Schuller]

So, these:

>  INFO [GC inspection] 2010-08-19 16:34:46,656 GCInspector.java (line 116) GC
> for ConcurrentMarkSweep: 41615 ms, 192522712 reclaimed leaving 8326856720
> used; max is 8700035072
[snip]
> INFO [GC inspection] 2010-08-19 16:36:00,786 GCInspector.java (line 116) GC 
> for ConcurrentMarkSweep: 37122 ms, 157488
> reclaimed leaving 8342836376 used; max is 8700035072

...show that you're live set is indeed very close to heap maximum, and
so concurrent mark/sweep phases run often freeing very little memory.
In addition the fact that it seems to take 35-45 seconds to do the
concurrent mark/sweep on an 8 gig heap on a modern system suggests
that you are probably CPU bound in cassandra at the time (meaning GC
is slower).

In short you're using too much memory in comparison to the maximum
heap size. The expected result is to either get an OOM, or just become
too slow due to excessive GC activity (usually the latter followed by
the former).

Now, the question is what memory is used *for*, and why. First off, to
get that out of the way, are you inserting with consistency level
ZERO? I am not sure whether it applies to 0.6.4 or not but there used
to be an issue involving writes at consistency level ZERO not being
throttled at all, meaning that if you threw writes at the system
faster than it would handle them, you would accumulate memory use. I
don't believe this is a problem with CL.ONE and above, even in 0.6.4
(but someone correct me if I'm wrong).

(As an aside: I'm not sure whether the behavior was such that it might
explain OOM on restart as a result of accumulated commitlogs that get
replayed faster than memtable flushing happens. Perhaps not, not
sure.)

In any case, the most important factors are what you're actually doing
with the cluster, but you don't say much about the data. In particular
how many rows and colums you're populating it with.

The primary users of large amounts of memory in cassandra include
(hopefully I'm not missing something major);

* bloom filters that are used to efficiently avoid doing I/O on
sstables that do not contain relevant data. the size of bloom filters
scale linearly with the number of row keys (not columns right? I don't
remember). so here we have an expected permanent, but low, memory use
as a result of a large database. how large is your database? 100
million keys? 1 billion? 10 billion?

* the memtables; the currently active memtable and any memtables
currently undergoing flushing. the size of these are directly
controllable in the configuration file. make sure they are reasonable.
(If you're not sure at all, with an 8 gig heap I'd say <= 512 mb is a
reasonable recommendation unless you have a reason to make them
larger)

* row cache and key cache, both controllable in the configuration. in
particular the row cache can be huge if you have configured it as
such.

* to some extent unflushed commitlogs; the commit log rotation
threshold controls this. the default value is low enough that it
should not be your culprit

So the question is what you're usage is like. How many unique rows do
you have? How many columns? The data size in and of itself should not
matter much to memory use, except of course that extremely large
individual values will be relevant to transient high memory use when
they are read/written.

In general, lacking large row caches and such things, you should be
able to have hundreds of millions of entries on an 8 gb heap, assuming
reasonably sized keys.

-- 
/ Peter Schuller

Re: Node OOM Problems

[Edward Capriolo]

On Thu, Aug 19, 2010 at 4:13 PM, Wayne  wrote:
> We are using the random partitioner. The tokens we defined manually and data
> is almost totally equal among nodes, 15GB per node when the trouble started.
> System vitals look fine. CPU load is ~500% for java, iostats are low,
> everything for all practical purposes looks "normal". Cfstats reports ~1-2
> ms for a read and ~.015-.02 for a write. All cache is now turned off as that
> seems to cause issues a lot faster. In terms of the current load, it is
> single insert statements. 6 nodes and 4 processes pointed to each node. That
> is not blasting away in my book.
>
> We can go a lot higher with memory, but with an 8g heap an NO cache thought
> this was a good number. With cache we would go higher on the memory but we
> are trying to get past this issue before even using cache. The possibility
> of a rogue large row is one I never considered. The largest row on the other
> nodes is as much as 800megs. I can not get a cfstats reading on the bad node
> so do not know how big its largest row is. I will raise memory to 16gb and
> see if that makes a difference. I had though that the java heap sizes that
> high had issues on their own in term of GC.
>
> Thanks.
>
>
> On Thu, Aug 19, 2010 at 9:44 PM, Edward Capriolo 
> wrote:
>>
>> On Thu, Aug 19, 2010 at 2:48 PM, Wayne  wrote:
>> > I am having some serious problems keeping a 6 node cluster up and
>> > running
>> > and stable under load. Any help would be greatly appreciated.
>> >
>> > Basically it always comes back to OOM errors that never seem to subside.
>> > After 5 minutes or 3 hours of heavy load depending on settings one or
>> > more
>> > nodes seem to go down with an OOM error. Upon restart the node tries to
>> > get
>> > going but ends up with an OOM error again and again and again. This
>> > patterns
>> > repeats and repeats with no way out. Once a node goes south on me
>> > deleting
>> > everything and starting from scratch is my only option. I can never get
>> > it
>> > to stay up again.
>> >
>> > I will try to describe the latest in detail. I am running a migration
>> > from a
>> > MySQL database with 24 processes against 6 nodes in python using the std
>> > thrift client. Should be a piece of cake in cassandra but no matter what
>> > I
>> > have tried it always seems to kill a node after not too long.
>> >
>> > 6 x Nodes with 8 cores, 8G heap, 24g physical
>> > 0.6.4
>> > JVM Options for lower compaction priority
>> > JVM 1.6.0_21-b06
>> >
>> > After 3 hours one node hard crashed with an OOM error. See below for
>> > messages.
>> >
>> > I restart and then see nothing but MessageDeserializationTask & some
>> > GCInspector messages.
>> >
>> >  INFO [GC inspection] 2010-08-19 17:06:09,228 GCInspector.java (line
>> > 143)
>> > MEMTABLE-POST-FLUSHER 1 1
>> >  WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-19 17:06:09,227
>> > MessageDeserializationTask.java (line 47) dropping message (52,288ms
>> > past
>> > timeout)
>> >  INFO [GC inspection] 2010-08-19 17:06:09,229 GCInspector.java (line
>> > 143)
>> > AE-SERVICE-STAGE  0 0
>> >  WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-19 17:06:09,229
>> > MessageDeserializationTask.java (line 47) dropping message (52,290ms
>> > past
>> > timeout)
>> >  INFO [GC inspection] 2010-08-19 17:06:09,229 GCInspector.java (line
>> > 143)
>> > FLUSH-WRITER-POOL 1 1
>> >
>> >
>> > After the 2nd restart it dies/freezes completely without actually
>> > crashing
>> > (kill -9 is required). I knew it was dead because nothing was logged for
>> > 30
>> > minutes to the log and nodetool still registered the node as down.
>> >
>> > After the 3rd reboot it comes back UP but then eventually crashes again.
>> > See
>> > below for details.
>> >
>> > I have tried using key or row cache but cache makes the problem happen
>> > in 5
>> > minutes as there is less memory and the OOM happens almost immediately.
>> > I
>> > have played with setting less concurrent readers and writers and now
>> > have it
>> > set to 16/16 (read/write). I have tried to set the mmap_index_only with
>> > no
>> > change as the issue is really a JVM issue. All other settings are
>> > std/default.
>> >
>> > Does anyone keep their nodes up under load over time? I have been
>> > working
>> > with cassandra for a while now and still have yet to keep anything up
>> > under
>> > load for very long. I know nothing about java, and frankly am starting
>> > to
>> > wonder if I need to be a java programmer to use use cassandra/run in
>> > debug
>> > mode. Any help would be greatly appreciated. These issues in the past
>> > made
>> > me try to use hbase which was solid as a rock (comparatively) but SLOW
>> > and
>> > too complicated...I came back again thinking .6.4 and soon .7 would be
>> > better but it almost seems worse to me in terms of stability. Frankly
>> > next I
>> > will have to look at other alternatives because cassandra is totally
>> > unstable. I do n

Re: Node OOM Problems

[Edward Capriolo]

On Thu, Aug 19, 2010 at 2:48 PM, Wayne  wrote:
> I am having some serious problems keeping a 6 node cluster up and running
> and stable under load. Any help would be greatly appreciated.
>
> Basically it always comes back to OOM errors that never seem to subside.
> After 5 minutes or 3 hours of heavy load depending on settings one or more
> nodes seem to go down with an OOM error. Upon restart the node tries to get
> going but ends up with an OOM error again and again and again. This patterns
> repeats and repeats with no way out. Once a node goes south on me deleting
> everything and starting from scratch is my only option. I can never get it
> to stay up again.
>
> I will try to describe the latest in detail. I am running a migration from a
> MySQL database with 24 processes against 6 nodes in python using the std
> thrift client. Should be a piece of cake in cassandra but no matter what I
> have tried it always seems to kill a node after not too long.
>
> 6 x Nodes with 8 cores, 8G heap, 24g physical
> 0.6.4
> JVM Options for lower compaction priority
> JVM 1.6.0_21-b06
>
> After 3 hours one node hard crashed with an OOM error. See below for
> messages.
>
> I restart and then see nothing but MessageDeserializationTask & some
> GCInspector messages.
>
>  INFO [GC inspection] 2010-08-19 17:06:09,228 GCInspector.java (line 143)
> MEMTABLE-POST-FLUSHER 1 1
>  WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-19 17:06:09,227
> MessageDeserializationTask.java (line 47) dropping message (52,288ms past
> timeout)
>  INFO [GC inspection] 2010-08-19 17:06:09,229 GCInspector.java (line 143)
> AE-SERVICE-STAGE  0 0
>  WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-19 17:06:09,229
> MessageDeserializationTask.java (line 47) dropping message (52,290ms past
> timeout)
>  INFO [GC inspection] 2010-08-19 17:06:09,229 GCInspector.java (line 143)
> FLUSH-WRITER-POOL 1 1
>
>
> After the 2nd restart it dies/freezes completely without actually crashing
> (kill -9 is required). I knew it was dead because nothing was logged for 30
> minutes to the log and nodetool still registered the node as down.
>
> After the 3rd reboot it comes back UP but then eventually crashes again. See
> below for details.
>
> I have tried using key or row cache but cache makes the problem happen in 5
> minutes as there is less memory and the OOM happens almost immediately. I
> have played with setting less concurrent readers and writers and now have it
> set to 16/16 (read/write). I have tried to set the mmap_index_only with no
> change as the issue is really a JVM issue. All other settings are
> std/default.
>
> Does anyone keep their nodes up under load over time? I have been working
> with cassandra for a while now and still have yet to keep anything up under
> load for very long. I know nothing about java, and frankly am starting to
> wonder if I need to be a java programmer to use use cassandra/run in debug
> mode. Any help would be greatly appreciated. These issues in the past made
> me try to use hbase which was solid as a rock (comparatively) but SLOW and
> too complicated...I came back again thinking .6.4 and soon .7 would be
> better but it almost seems worse to me in terms of stability. Frankly next I
> will have to look at other alternatives because cassandra is totally
> unstable. I do not want to give up after all this time but I am out of
> ideas.
>
> HELP!
>
> Messages prior to first crash:
>
>  WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-19 16:33:19,985
> MessageDeserializationTask.java (line 47) dropping message (12,024ms past
> timeout)
>  INFO [GC inspection] 2010-08-19 16:33:19,985 GCInspector.java (line 143)
> CONSISTENCY-MANAGER   0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,986 GCInspector.java (line 143)
> LB-TARGET 0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,986 GCInspector.java (line 143)
> ROW-MUTATION-STAGE    1   167
>  INFO [GC inspection] 2010-08-19 16:33:19,987 GCInspector.java (line 143)
> MESSAGE-STREAMING-POOL    0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,987 GCInspector.java (line 143)
> LOAD-BALANCER-STAGE   0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,987 GCInspector.java (line 143)
> FLUSH-SORTER-POOL 0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,987 GCInspector.java (line 143)
> MEMTABLE-POST-FLUSHER 0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,987 GCInspector.java (line 143)
> AE-SERVICE-STAGE  0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,988 GCInspector.java (line 143)
> FLUSH-WRITER-POOL 0 0
>  INFO [GC inspection] 2010-08-19 16:33:19,988 GCInspector.java (line 143)
> HINTED-HANDOFF-POOL   1 1
>  INFO [GC inspection] 2010-08-19 16:33:19,988 GCInspector.java (line 148)
> CompactionManager   n/a 0
> 

Re: Cassandra disk space utilization WAY higher than I would expect

[Robert Coli]

On Thu, Aug 19, 2010 at 7:23 AM, Julie  wrote:
> At this point, I logged in.  The data distribution on this node was 122GB.  I
> started performing a manual nodetool cleanup.

Check the size of the Hinted Handoff CF? If your nodes are flapping
under sustained write, they could be storing a non-trivial number of
hinted handoff rows? Probably not 5x usage though..

http://wiki.apache.org/cassandra/Operations
"
The reason why you run nodetool cleanup on all live nodes [after
replacing a node] is to remove old Hinted Handoff writes stored for
the dead node.
"

You could relatively quickly determine whether Hinted Handoff is
implicated by running your test with the feature turned off.

https://issues.apache.org/jira/browse/CASSANDRA-894

=Rob

Re: Cassandra w/ Hadoop

[Mark]

 On 8/19/10 10:23 AM, Jeremy Hanna wrote:

I would check out http://wiki.apache.org/cassandra/HadoopSupport for more info. 
 I'll try to explain a bit more here, but I don't think there's a tutorial out 
there yet.

For input:
- configure your main class where you're starting the mapreduce job the way the 
word_count is configured (with either storage-conf or in your code via the 
ConfigHelper).  It will complain specifically about stuff you hadn't configured 
- esp. important is your cassandra server and port.
- the inputs to your mapper are going to be what's coming from cassandra - so 
your key with a map of row values
- you need to set your column name in your overridden setup method in your 
mapper
- for the reducer, nothing really changes from a normal map/reduce, unless you 
want to output to cassandra
- generally cassandra just provides an inputformat and split classes to read 
from cassandra - you can find the guts in the org.apache.cassandra.hadoop 
package

For output:
- in your reducer, you could just write to cassandra directly via thrift.  
there is a built-in outputformat coming in 0.7 but it still might change before 
0.7 final - that will queue up changes so it will write large blocks all at 
once.


On Aug 19, 2010, at 12:07 PM, Mark wrote:


Are there any examples/tutorials on the web for reading/writing from Cassandra 
into/from Hadoop?

I found the example in contrib/word_count but I really can't make sense of 
it... a tutorial/explanation would help.

Thanks!

Re: Cassandra w/ Hadoop

[Mark]

 On 8/19/10 10:34 AM, Christian Decker wrote:
If, like me, you prefer to write your jobs on the fly try taking a 
look at Pig. Cassandra provides a loadfunc under contrib/pig/ in the 
source package which allows you to load data directly from Cassandra.

--
Christian Decker
Software Architect
http://blog.snyke.net


On Thu, Aug 19, 2010 at 7:23 PM, Jeremy Hanna 
mailto:jeremy.hanna1...@gmail.com>> wrote:


I would check out http://wiki.apache.org/cassandra/HadoopSupport
for more info.  I'll try to explain a bit more here, but I don't
think there's a tutorial out there yet.

For input:
- configure your main class where you're starting the mapreduce
job the way the word_count is configured (with either storage-conf
or in your code via the ConfigHelper).  It will complain
specifically about stuff you hadn't configured - esp. important is
your cassandra server and port.
- the inputs to your mapper are going to be what's coming from
cassandra - so your key with a map of row values
- you need to set your column name in your overridden setup method
in your mapper
- for the reducer, nothing really changes from a normal
map/reduce, unless you want to output to cassandra
- generally cassandra just provides an inputformat and split
classes to read from cassandra - you can find the guts in the
org.apache.cassandra.hadoop package

For output:
- in your reducer, you could just write to cassandra directly via
thrift.  there is a built-in outputformat coming in 0.7 but it
still might change before 0.7 final - that will queue up changes
so it will write large blocks all at once.


On Aug 19, 2010, at 12:07 PM, Mark wrote:

> Are there any examples/tutorials on the web for reading/writing
from Cassandra into/from Hadoop?
>
> I found the example in contrib/word_count but I really can't
make sense of it... a tutorial/explanation would help.


That's definitely an option and I'll probably lean towards that in the 
near future. I am just trying to get a complete understanding of the 
whole infrastructure before working with higher level features.


Also same problem exists... I need a nice tutorial :)

Re: Cassandra w/ Hadoop

[Christian Decker]

If, like me, you prefer to write your jobs on the fly try taking a look at
Pig. Cassandra provides a loadfunc under contrib/pig/ in the source package
which allows you to load data directly from Cassandra.
--
Christian Decker
Software Architect
http://blog.snyke.net


On Thu, Aug 19, 2010 at 7:23 PM, Jeremy Hanna wrote:

> I would check out http://wiki.apache.org/cassandra/HadoopSupport for more
> info.  I'll try to explain a bit more here, but I don't think there's a
> tutorial out there yet.
>
> For input:
> - configure your main class where you're starting the mapreduce job the way
> the word_count is configured (with either storage-conf or in your code via
> the ConfigHelper).  It will complain specifically about stuff you hadn't
> configured - esp. important is your cassandra server and port.
> - the inputs to your mapper are going to be what's coming from cassandra -
> so your key with a map of row values
> - you need to set your column name in your overridden setup method in your
> mapper
> - for the reducer, nothing really changes from a normal map/reduce, unless
> you want to output to cassandra
> - generally cassandra just provides an inputformat and split classes to
> read from cassandra - you can find the guts in the
> org.apache.cassandra.hadoop package
>
> For output:
> - in your reducer, you could just write to cassandra directly via thrift.
>  there is a built-in outputformat coming in 0.7 but it still might change
> before 0.7 final - that will queue up changes so it will write large blocks
> all at once.
>
>
> On Aug 19, 2010, at 12:07 PM, Mark wrote:
>
> > Are there any examples/tutorials on the web for reading/writing from
> Cassandra into/from Hadoop?
> >
> > I found the example in contrib/word_count but I really can't make sense
> of it... a tutorial/explanation would help.
>
>

Re: Thrift + PHP: help!

[Dave Viner]

I am a user of the perl api - so I'd like to lurk in case there are things
that can benefit both perl & php.

Dave Viner


On Wed, Aug 18, 2010 at 1:35 PM, Gabriel Sosa  wrote:

> I would like to help with this too!
>
>
> On Wed, Aug 18, 2010 at 5:15 PM, Bas Kok  wrote:
>
>> I have some experience in this area and would be happy to help out as
>> well.
>> --
>> Bas
>>
>>
>> On Wed, Aug 18, 2010 at 8:26 PM, Dave Gardner 
>> wrote:
>>
>>> I'm happy to assist. Having a robust PHP implementation would help us
>>> greatly.
>>>
>>> Dave
>>>
>>> On Wednesday, August 18, 2010, Jeremy Hanna 
>>> wrote:
>>> > As Jonathan mentioned in his keynote at the Cassandra Summit, the
>>> thrift + php has some bugs and is maintainerless right now.
>>> >
>>> > Is there anyone out there in the Cassandra community that is adept at
>>> PHP that could help out with the thrift + php work?  It would benefit all
>>> who use Cassandra with PHP.
>>> >
>>> > Bryan Duxbury, a thrift developer/committer, said "if someone really
>>> wanted to have a go at making thrift php robust, i would assist them
>>> heavily."
>>> >
>>> > Please respond to this thread or ask Bryan in the channel.
>>>
>>
>>
>
>
> --
> Gabriel Sosa
> Si buscas resultados distintos, no hagas siempre lo mismo. - Einstein
>

Re: Cassandra w/ Hadoop

[Jeremy Hanna]

I would check out http://wiki.apache.org/cassandra/HadoopSupport for more info. 
 I'll try to explain a bit more here, but I don't think there's a tutorial out 
there yet.

For input:
- configure your main class where you're starting the mapreduce job the way the 
word_count is configured (with either storage-conf or in your code via the 
ConfigHelper).  It will complain specifically about stuff you hadn't configured 
- esp. important is your cassandra server and port.
- the inputs to your mapper are going to be what's coming from cassandra - so 
your key with a map of row values
- you need to set your column name in your overridden setup method in your 
mapper
- for the reducer, nothing really changes from a normal map/reduce, unless you 
want to output to cassandra
- generally cassandra just provides an inputformat and split classes to read 
from cassandra - you can find the guts in the org.apache.cassandra.hadoop 
package

For output:
- in your reducer, you could just write to cassandra directly via thrift.  
there is a built-in outputformat coming in 0.7 but it still might change before 
0.7 final - that will queue up changes so it will write large blocks all at 
once.


On Aug 19, 2010, at 12:07 PM, Mark wrote:

> Are there any examples/tutorials on the web for reading/writing from 
> Cassandra into/from Hadoop?
> 
> I found the example in contrib/word_count but I really can't make sense of 
> it... a tutorial/explanation would help.

Cassandra w/ Hadoop

[Mark]

 Are there any examples/tutorials on the web for reading/writing from 
Cassandra into/from Hadoop?


I found the example in contrib/word_count but I really can't make sense 
of it... a tutorial/explanation would help.

Re: Pig + Cassandra = Connection errors

[Christian Decker]

In the hopes to better understand the problem I took the liberty of putting
the storage-conf.xml on the net [1]. I even tried starting from scratch
again, and taking care about which interfaces I use, and what ports I bind
to, but until now, nothing really got me anywhere.

[1] http://pastebin.com/xeVkAkHu
--
Christian Decker
Software Architect
http://blog.snyke.net

Re: Data Corruption

[Jonathan Ellis]

You're moving data around manually?  That sounds like a good way to
confuse Cassandra's replication.

On Thu, Aug 19, 2010 at 4:33 AM, Waqas Badar
 wrote:
> We are observing a strange behavior of Cassandra. We have a ring of two
> nodes. When we inserts data in cassandra then old data after some entries
> get vanished. Please note that it is not a data loss, as when we move that
> data to separate node then all data is shown.  We are using Cassandra 0.6.3
> and Ubuntu Server 8.10



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: kundera: Open source JPA 1.0 compliant ORM for Cassandra

[Chen Xinli]

I have a question about lucandra: the row-ids from reversed-index is
computed in cassandra , or using a client to get reversed index first and
computed in client machine?

2010/8/19 Animesh Kumar 

> Hi Jonathan,
>
> Glad that you liked the effort. Kundera team is constantly watching
> Cassandra developments to pick features up as soon as they become available,
> for example, secondary indexes, schema generator support etc.
>
> I would like to highlight few important stuffs about Kundera:
>
> 1. Kundera has been architected so as to be able to use any existing
> Cassandra client library, Hector, Pelops whatever, in simple adaptor design
> format. Kundera though supports Pelops only, however should anyone want to
> write a custom adaptor for any of his favorite client libraries he can
> easily do it.
>
> 2. Kundera - for now - uses Lucandra to store reverse-indexes into
> Cassandra itself. Support for Solr will be introduced very soon.
>
> 3. Kundera supports JPA defined relationships like, OneToOne, OneToMany,
> ManyToOne and ManyToMany between concrete entity objects with full Cascade
> support and Lazy/Eager loading strategies. This is achieved by storing
> foreign row-ids into holding entity's CF row. Kundera doesn't support entity
> embedding or entity inheritance yet, but they are in pipeline.
>
> 4. Kundera converts JPA queries to Lucene queries which run on Lucene
> indexes to fetch row-ids which are used to look-up data from Cassandra CFs.
> You can also directly execute Lucene queries.
>
> This might have given you a better picture of Kundera, should you need more
> information you can visit: http://kundera.googlecode.com
>
> Thanks,
> Animesh
> iLabs, Impetus
>
>
>
> -Original Message-
> From: Jonathan Ellis [mailto:jbel...@gmail.com]
> Sent: Sunday, August 01, 2010 11:01 PM
> To: user@cassandra.apache.org
> Subject: Re: kundera: Open source JPA 1.0 compliant ORM for Cassandra
>
> Glad to see this, and I hope you'll be able to take advantage of the
> secondary index features coming in 0.7.
>
> Supporting different lower-level "drivers" seems like wasted effort to me,
> fwiw.
>
> On Sat, Jul 31, 2010 at 8:54 AM, Sanjay Sharma
>  wrote:
> > Hi All,
> >
> > We are happy to announce and share a new ORM over Cassandra - kundera
> >
> > The project is Apache licensed and hosted at
> http://kundera.googlecode.com
> >
> >
> >
> > The project uses custom Cassandra Annotations and is fully JPA 1.0
> > compliant. @ColumnFamily and @SuperColumnFamily are the main Cassandra
> > specific annotations.
> >
> >
> >
> > Search/Indexing is automatically included by using "Lucandra" and drives
> the
> > JPA-QL query support. Use of Lucandra also enables users to write Lucene
> > queries along with JPA-QL queries.
> >
> >
> >
> > As per the main author of kundera - Animesh -" The idea behind Kundera is
> to
> > make working with Cassandra drop-dead simple and fun. Kundera does not
> > reinvent the wheel by making another client library; rather it leverages
> the
> > existing libraries and builds - on top of them - a wrap-around API to
> help
> > developers do away with unnecessary boiler plate codes, and program a
> > neater-and-cleaner code that reduces code-complexity and improves
> quality.
> > And above all, improves productivity."
> >
> >
> >
> > The current implementation uses the versatile "Pelops" library as the
> > underlying client API and plans are to add support for Hector and Thrift
> > clients as well.
> >
> >
> >
> > Here is a sample kundera Entity bean -
> >
> > @Entity
> >
> > @ColumnFamily(keyspace = "Keyspace1", family = "SimpleComment")
> >
> > public class SimpleComment {
> >
> > @Id
> >
> > private String id;
> >
> > @Column(name = "userId")
> >
> > private String userId;
> >
> > @Column(name = "comment")
> >
> > private String commentText;
> >
> >
> >
> > ..
> >
> > }
> >
> > JPA queries are as simple as-
> >
> > Query query = entityManager.createQuery("SELECT c from
> SimpleComment
> > c where userId='me'");
> >
> > List list = query.getResultList();
> >
> >
> >
> > There is already support for Spring based persistence integration like
> the
> > good old Spring+Hibernate integration and is as simple as this-
> >
> >  >
> class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
> >
> > 
> >
> > 
> >
> > More examples are available in kundera's wiki and Animesh's blogs. The
> > spring integration example is here
> >
> >
> >
> > Regards,
> >
> > Sanjay Sharma
> >
> > iLabs, Impetus
> >
> > 
> > Impetus is sponsoring 'Hadoop India User Group Meet Up'- a technology
> > un-conference on July 31, 2010 at Impetus Office, Noida. The event will
> shed
> > light on Hadoop technology and channelized efforts to develop an active
> > Hadoop community.
> >
> > Click http://www.impetus.com/ to know more. Follow our updates on
> > www.twitter.com/impetuscalling .
> >
> >
> > NO

Re: Cassandra disk space utilization WAY higher than I would expect

[Julie]

Peter Schuller  infidyne.com> writes:

> Without necessarily dumping all the information - approximately what
> do they contain? Do they contain anything about compactions,
> anti-compactions, streaming, etc?
> 
> With an idle node after taking writes, I *think* the only expected
> disk I/O (once it has settled) would be a memtable flush triggered by
> memtable_flush_after_mins, and possibly compactions resulting from
> that (depending on how close one were to triggering compaction prior
> to the memtable flush). Whatever is causing additional sstables to be
> written, even if somehow triggered incorrectly, I'd hope that they
> were logged still.

**
Hi Peter,

Thanks for your ideas.  Here is what I see in the bloated node's system.log file
(condensed) starting 4 minutes after the last write.  At this time, data usage
on this node was only around 28GB (as expected).
**

INFO [COMPACTION-POOL:1] 2010-08-16 13:51:38,432 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 14:20:02,840 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 14:28:37,085 CompactionManager.java
 (line 246) Compacting ...

 INFO [Thread-25] 2010-08-16 14:34:06,827 StreamCompletionHandler.java
 (line 64) Streaming added
 /var/lib/cassandra/data/Keyspace1/Standard1-467-Data.db

 INFO [Timer-1] 2010-08-16 14:34:31,655 Gossiper.java (line 179)
 InetAddress /10.254.107.178 is now dead.
 INFO [GMFD:1] 2010-08-16 14:35:03,012 Gossiper.java (line 568)
 InetAddress /10.254.107.178 is now UP

 INFO [COMPACTION-POOL:1] 2010-08-16 14:36:04,808 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 14:40:31,283 CompactionManager.java
 (line 246) Compacting ...
 INFO [FLUSH-TIMER] 2010-08-16 14:50:46,570 ColumnFamilyStore.java (line 357)
 Standard1 has reached its threshold; switching in a fresh Memtable at
 CommitLogContext(file=
 '/var/lib/cassandra/commitlog/CommitLog-1281980828674.log', position=87493038)


 INFO [FLUSH-TIMER] 2010-08-16 14:50:46,602 ColumnFamilyStore.java (line 609)
 Enqueuing flush of Memtable(Standard1)@31873581
 INFO [FLUSH-WRITER-POOL:1] 2010-08-16 14:50:46,647 Memtable.java (line 148)
 Writing Memtable(Standard1)@31873581
 INFO [GC inspection] 2010-08-16 14:50:47,350 GCInspector.java (line 110)
 GC for ConcurrentMarkSweep: 362 ms, 506927464 reclaimed leaving 129128920 used;
 max is 1172766720
 INFO [FLUSH-WRITER-POOL:1] 2010-08-16 14:50:51,137 Memtable.java (line 162)
 Completed flushing /var/lib/cassandra/data/Keyspace1/Standard1-472-Data.db
 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:50:58,411
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-223-Data.db
...
...
 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:51:40,662
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-407-Data.db

 INFO [Timer-1] 2010-08-16 14:51:40,772 Gossiper.java (line 179)
 InetAddress /10.254.242.159 is now dead.

 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:51:41,011
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-279-Data.db
 ...
 ...
 INFO [SSTABLE-CLEANUP-TIMER] 2010-08-16 14:52:22,206
 SSTableDeletingReference.java (line 104)
 Deleted /var/lib/cassandra/data/Keyspace1/Standard1-389-Data.db

 INFO [GMFD:1] 2010-08-16 14:52:34,906 Gossiper.java (line 568)
 InetAddress /10.254.242.159 is now UP

 INFO [COMPACTION-POOL:1] 2010-08-16 14:58:48,958 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 15:02:43,444 CompactionManager.java
 (line 246) Compacting ...
 INFO [COMPACTION-POOL:1] 2010-08-16 15:05:08,837 CompactionManager.java
 (line 246) Compacting []

 INFO [Timer-1] 2010-08-16 15:17:38,697 Gossiper.java (line 179)
 InetAddress /10.254.234.226 is now dead.
 INFO [GMFD:1] 2010-08-16 15:18:37,471 Gossiper.java (line 568)
 InetAddress /10.254.234.226 is now UP


 INFO [Thread-28] 2010-08-16 15:38:34,685 StreamCompletionHandler.java
 (line 64)
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-475-Data.db

 INFO [COMPACTION-POOL:1] 2010-08-16 16:05:08,811 CompactionManager.java
 (line 246) Compacting []
 INFO [COMPACTION-POOL:1] 2010-08-16 17:05:08,792 CompactionManager.java
 (line 246) Compacting []

 INFO [Timer-1] 2010-08-16 17:05:55,583 Gossiper.java (line 179)
 InetAddress /10.254.242.159 is now dead.
 INFO [GMFD:1] 2010-08-16 17:06:19,614 Gossiper.java (line 568)
 InetAddress /10.254.242.159 is now UP


 INFO [Thread-34] 2010-08-16 17:24:57,584 StreamCompletionHandler.java
 (line 64)
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-477-Data.db

 INFO [Timer-1] 2010-08-16 17:25:19,206 Gossiper.java (line 179)
 InetAddress /10.210.157.187 is now dead.
 INFO [GMFD:1] 2010-08-16 17:25:53,096 Gossiper.java (line 568)
 InetAddress /10.210.157.187 is now UP

Data Corruption

[Waqas Badar]

We are observing a strange behavior of Cassandra. We have a ring of two
nodes. When we inserts data in cassandra then old data after some
entries get vanished. Please note that it is not a data loss, as when we
move that data to separate node then all data is shown.  We are using
Cassandra 0.6.3 and Ubuntu Server 8.10

RE: kundera: Open source JPA 1.0 compliant ORM for Cassandra

[Animesh Kumar]

Hi Jonathan,

Glad that you liked the effort. Kundera team is constantly watching Cassandra 
developments to pick features up as soon as they become available, for example, 
secondary indexes, schema generator support etc.

I would like to highlight few important stuffs about Kundera:

1. Kundera has been architected so as to be able to use any existing Cassandra 
client library, Hector, Pelops whatever, in simple adaptor design format. 
Kundera though supports Pelops only, however should anyone want to write a 
custom adaptor for any of his favorite client libraries he can easily do it.

2. Kundera - for now - uses Lucandra to store reverse-indexes into Cassandra 
itself. Support for Solr will be introduced very soon.

3. Kundera supports JPA defined relationships like, OneToOne, OneToMany, 
ManyToOne and ManyToMany between concrete entity objects with full Cascade 
support and Lazy/Eager loading strategies. This is achieved by storing foreign 
row-ids into holding entity's CF row. Kundera doesn't support entity embedding 
or entity inheritance yet, but they are in pipeline.

4. Kundera converts JPA queries to Lucene queries which run on Lucene indexes 
to fetch row-ids which are used to look-up data from Cassandra CFs. You can 
also directly execute Lucene queries.

This might have given you a better picture of Kundera, should you need more 
information you can visit: http://kundera.googlecode.com

Thanks,
Animesh
iLabs, Impetus



-Original Message-
From: Jonathan Ellis [mailto:jbel...@gmail.com]
Sent: Sunday, August 01, 2010 11:01 PM
To: user@cassandra.apache.org
Subject: Re: kundera: Open source JPA 1.0 compliant ORM for Cassandra

Glad to see this, and I hope you'll be able to take advantage of the
secondary index features coming in 0.7.

Supporting different lower-level "drivers" seems like wasted effort to me, fwiw.

On Sat, Jul 31, 2010 at 8:54 AM, Sanjay Sharma
 wrote:
> Hi All,
>
> We are happy to announce and share a new ORM over Cassandra - kundera
>
> The project is Apache licensed and hosted at http://kundera.googlecode.com
>
>
>
> The project uses custom Cassandra Annotations and is fully JPA 1.0
> compliant. @ColumnFamily and @SuperColumnFamily are the main Cassandra
> specific annotations.
>
>
>
> Search/Indexing is automatically included by using "Lucandra" and drives the
> JPA-QL query support. Use of Lucandra also enables users to write Lucene
> queries along with JPA-QL queries.
>
>
>
> As per the main author of kundera - Animesh -" The idea behind Kundera is to
> make working with Cassandra drop-dead simple and fun. Kundera does not
> reinvent the wheel by making another client library; rather it leverages the
> existing libraries and builds - on top of them - a wrap-around API to help
> developers do away with unnecessary boiler plate codes, and program a
> neater-and-cleaner code that reduces code-complexity and improves quality.
> And above all, improves productivity."
>
>
>
> The current implementation uses the versatile "Pelops" library as the
> underlying client API and plans are to add support for Hector and Thrift
> clients as well.
>
>
>
> Here is a sample kundera Entity bean -
>
> @Entity
>
> @ColumnFamily(keyspace = "Keyspace1", family = "SimpleComment")
>
> public class SimpleComment {
>
> @Id
>
> private String id;
>
> @Column(name = "userId")
>
> private String userId;
>
> @Column(name = "comment")
>
> private String commentText;
>
>
>
> ..
>
> }
>
> JPA queries are as simple as-
>
> Query query = entityManager.createQuery("SELECT c from SimpleComment
> c where userId='me'");
>
> List list = query.getResultList();
>
>
>
> There is already support for Spring based persistence integration like the
> good old Spring+Hibernate integration and is as simple as this-
>
>  class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
>
> 
>
> 
>
> More examples are available in kundera's wiki and Animesh's blogs. The
> spring integration example is here
>
>
>
> Regards,
>
> Sanjay Sharma
>
> iLabs, Impetus
>
> 
> Impetus is sponsoring 'Hadoop India User Group Meet Up'- a technology
> un-conference on July 31, 2010 at Impetus Office, Noida. The event will shed
> light on Hadoop technology and channelized efforts to develop an active
> Hadoop community.
>
> Click http://www.impetus.com/ to know more. Follow our updates on
> www.twitter.com/impetuscalling .
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interfe

Re: Help with getting Key range with some column limitations

[Chen Xinli]

Hi,

If reading latency is tolerable, you can get 700 columns every time, set end
key of last iteration as start key of next iteration, to retrieve all the
data.

Or you can implement a plugin of cassandra to do column filter, only returns
the data you want.
The computation is done locally in cassandra machine, and will be quite fast
to avoid timeoutexception.

2010/8/19 Jone Lura 

> Hi,
>
> We are trying to implement Cassandra to replace one of our biggest SQL
> tables, and so far we got it working.
>
> However, for testing I'm using Cassandra 0.6.2, Java and Pelops. (Pelops
> not that important for my question) and need suggestions on how to solve a
> problem retrieving a key range based on the following.
>
> 
>  ColumnType="Super"
> CompareWith="LongType"
> KeysCached="100%"
> CompareSubcolumnsWith="UTF8Type" />
>  ...
> 
>
> The super column got columns for longitude and latitude.
>
>  1. Need to get get max long number for key
>  2. The key should also have supercolumns latitude and longitude column
> intersecting (inside) with a given boundingbox.
>
> Currently I'm doing like this
>
>
> KeyRange keyRange = new KeyRange();
> keyRange.setStart_key("");
> keyRange.setEnd_key("");
> keyRange.setCount(700);
>
> And checking every row in db if it match my bounding box.
>
> But there are a lot more than 700 keys.. and if i set a higher count, the
> get_range_slice get a Timeout Exception.
>
> Any ideas?
>
> Best Regards
> Jone
>



-- 
Best Regards,
Chen Xinli