Re: StackOverflowError on high load

[Jonathan Ellis]

looks like test1 started gc storming, so test2 treats it as dead and
starts doing hinted handoff for it, which increases test2's load, even
though test1 is not completely dead yet.

On Thu, Feb 18, 2010 at 1:16 AM, Ran Tavory  wrote:
> I found another interesting graph, attached.
> I looked at the write-count and write-latency of the CF I'm writing to and I
> see a few interesting things:
> 1. the host test2 crashed at 18:00
> 2. At 16:00, after a few hours of load both hosts dropped their write-count.
> test1 (which did not crash) started slowing down first and then test2
> slowed.
> 3. At 16:00 I start seeing high write-latency on test2 only. This takes
> about 2h until finally at 18:00 it crashes.
> Does this help?
>
> On Thu, Feb 18, 2010 at 7:44 AM, Ran Tavory  wrote:
>>
>> I ran the process again and after a few hours the same node crashed the
>> same way. Now I can tell for sure this is indeed what Jonathan proposed -
>> the data directory needs to be 2x of what it is, but it looks like a design
>> problem, how large to I need to tell my admin to set it then?
>> Here's what I see when the server crashes:
>> $ df -h /outbrain/cassandra/data/
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/mapper/cassandra-data
>>                        97G   46G   47G  50% /outbrain/cassandra/data
>> The directory is 97G and when the host crashes it's at 50% use.
>> I'm also monitoring various JMX counters and I see that COMPACTION-POOL
>> PendingTasks grows for a while on this host (not on the other host, btw,
>> which is fine, just this host) and then flats for 3 hours. After 3 hours of
>> flat it crashes. I'm attaching the graph.
>> When I restart cassandra on this host (not changed file allocation size,
>> just restart) it does manage to compact the data files pretty fast, so after
>> a minute I get 12% use, so I wonder what made it crash before that doesn't
>> now? (could be the load that's not running now)
>> $ df -h /outbrain/cassandra/data/
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/mapper/cassandra-data
>>                        97G   11G   82G  12% /outbrain/cassandra/data
>> The question is what size does the data directory need to be? It's not 2x
>> the size of the data I expect to have (I only have 11G of real data after
>> compaction and the dir is 97G, so it should have been enough). If it's 2x of
>> something dynamic that keeps growing and isn't bound then it'll just
>> grow infinitely, right? What's the bound?
>> Alternatively, what jmx counter thresholds are the best indicators for the
>> crash that's about to happen?
>> Thanks
>>
>> On Wed, Feb 17, 2010 at 9:00 PM, Tatu Saloranta 
>> wrote:
>>>
>>> On Wed, Feb 17, 2010 at 6:40 AM, Ran Tavory  wrote:
>>> > If it's the data directory, then I have a pretty big one. Maybe it's
>>> > something else
>>> > $ df -h /outbrain/cassandra/data/
>>> > Filesystem            Size  Used Avail Use% Mounted on
>>> > /dev/mapper/cassandra-data
>>> >                        97G   11G   82G  12% /outbrain/cassandra/data
>>>
>>> Perhaps a temporary file? JVM defaults to /tmp, which may be on a
>>> smaller (root) partition?
>>>
>>> -+ Tatu +-
>>
>
>

Re: StackOverflowError on high load

[Jonathan Ellis]

https://issues.apache.org/jira/browse/CASSANDRA-804 should have fixed
this in trunk / 0.6.  or at least log more about what is going on so
we can fix it better. :)

On Thu, Feb 18, 2010 at 12:44 AM, Ran Tavory  wrote:
> I ran the process again and after a few hours the same node crashed the same
> way. Now I can tell for sure this is indeed what Jonathan proposed - the
> data directory needs to be 2x of what it is, but it looks like a design
> problem, how large to I need to tell my admin to set it then?
> Here's what I see when the server crashes:
> $ df -h /outbrain/cassandra/data/
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/cassandra-data
>                        97G   46G   47G  50% /outbrain/cassandra/data
> The directory is 97G and when the host crashes it's at 50% use.
> I'm also monitoring various JMX counters and I see that COMPACTION-POOL
> PendingTasks grows for a while on this host (not on the other host, btw,
> which is fine, just this host) and then flats for 3 hours. After 3 hours of
> flat it crashes. I'm attaching the graph.
> When I restart cassandra on this host (not changed file allocation size,
> just restart) it does manage to compact the data files pretty fast, so after
> a minute I get 12% use, so I wonder what made it crash before that doesn't
> now? (could be the load that's not running now)
> $ df -h /outbrain/cassandra/data/
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/cassandra-data
>                        97G   11G   82G  12% /outbrain/cassandra/data
> The question is what size does the data directory need to be? It's not 2x
> the size of the data I expect to have (I only have 11G of real data after
> compaction and the dir is 97G, so it should have been enough). If it's 2x of
> something dynamic that keeps growing and isn't bound then it'll just
> grow infinitely, right? What's the bound?
> Alternatively, what jmx counter thresholds are the best indicators for the
> crash that's about to happen?
> Thanks
>
> On Wed, Feb 17, 2010 at 9:00 PM, Tatu Saloranta 
> wrote:
>>
>> On Wed, Feb 17, 2010 at 6:40 AM, Ran Tavory  wrote:
>> > If it's the data directory, then I have a pretty big one. Maybe it's
>> > something else
>> > $ df -h /outbrain/cassandra/data/
>> > Filesystem            Size  Used Avail Use% Mounted on
>> > /dev/mapper/cassandra-data
>> >                        97G   11G   82G  12% /outbrain/cassandra/data
>>
>> Perhaps a temporary file? JVM defaults to /tmp, which may be on a
>> smaller (root) partition?
>>
>> -+ Tatu +-
>
>

Re: cassandra freezes

[Santal Li]

the GC options as bellow:

JVM_OPTS=" \
-ea \
-Xms2G \
-Xmx8G \
-XX:SurvivorRatio=8 \
-XX:TargetSurvivorRatio=90 \
-XX:+AggressiveOpts \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:SurvivorRatio=128 \
-XX:MaxTenuringThreshold=0 \
-Dcom.sun.management.jmxremote.port=8080 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"


Regards
-Santal


2010/2/20 Jonathan Ellis 

> are you using the old deb package?  because that had broken gc settings.
>
> On Fri, Feb 19, 2010 at 10:40 PM, Santal Li  wrote:
> > I meet almost same thing as you. When I do some benchmarks write test,
> some
> > times one Cassandra will freeze and other node will consider it was
> shutdown
> > and up after 30+ second. I am using 5 node, each node 8G mem for java
> heap.
> >
> > From my investigate, it was caused by GC thread, because I start the
> > JConsole and monitor with the memory heap usage, each time when the GC
> > happend, heap usage will drop down from 6G to 1G, and check the casandra
> > log, I found the freeze happend at exactly same times.
> >
> > So I think when using huge memory(>2G), maybe need using some different
> GC
> > stratege other than the default one provide by Cassandra lunch script.
> > Dose't anyone meet this situation, can you please provide some guide?
> >
> >
> > Thanks
> > -Santal
> >
> > 2010/2/17 Tatu Saloranta 
> >>
> >> On Tue, Feb 16, 2010 at 6:25 AM, Boris Shulman 
> wrote:
> >> > Hello, I'm running some benchmarks on 2 cassandra nodes each running
> >> > on 8 cores machine with 16G RAM, 10G for Java heap. I've noticed that
> >> > during benchmarks with numerous writes cassandra just freeze for
> >> > several minutes (in those benchmarks I'm writing batches of 10 columns
> >> > with 1K data each for every key in a single CF). Usually after
> >> > performing 50K writes I'm getting a TimeOutException and cassandra
> >> > just freezes. What configuration changes can I make in order to
> >> > prevent this? Is it possible that my setup just can't handle the load?
> >> > How can I calculate the number of casandra nodes for a desired load?
> >>
> >> One thing that can cause seeming lockups is garbage collector. So
> >> enabling GC debug output would be heplful, to see GC activity. Some
> >> collector (CMS specifically) can stop the system for very long time,
> >> up to minutes. This is not necessarily the root cause, but is easy to
> >> rule out.
> >> Beyond this, getting a stack trace during lockup would make sense.
> >> That can pinpoint what threads are doing, or what they are blocked on
> >> in case there is a deadlock or heavy contention on some shared
> >> resource.
> >>
> >> -+ Tatu +-
> >
> >
>

PHP/TBinaryProtocolAccelerated I64 timestamp (microtime) issue

[Michael Pearson]

Hi, is anyone else using thrift/php TBinaryProtocolAccelerated
(thrift_protocol.so) call?  It doesn't look to be sending timestamps
correctly (casting as signed int 32)  no such issue with
TBinaryProtocol.

eg:

generating 13 digit cassandra_Column->timestamp via microtime()

insert() via TBinaryProtocol :

  cassandra> get Keyspace1.Standard1['PandraTest_Dim3']
  => (column=column1, value=TEST DATA, timestamp=1266635621757)

... which is fine

insert() via TBinaryProtocolAccelerated/thrift_protocol_write_binary  :

  cassandra> get Keyspace1.Standard1['PandraTest_Dim33']
  => (column=column1, value=TEST DATA, timestamp=-379667704)

... using different keys there, as the 'new' timestamp written with
thrift_protocol_write_binary is negative and therefore never seen.


Keen to see if anyone else has experienced this behaviour, I can
confirm most recent thrift has been pulled and installed from svn.

thanks

-michael

Re: cassandra freezes

[Jonathan Ellis]

are you using the old deb package?  because that had broken gc settings.

On Fri, Feb 19, 2010 at 10:40 PM, Santal Li  wrote:
> I meet almost same thing as you. When I do some benchmarks write test, some
> times one Cassandra will freeze and other node will consider it was shutdown
> and up after 30+ second. I am using 5 node, each node 8G mem for java heap.
>
> From my investigate, it was caused by GC thread, because I start the
> JConsole and monitor with the memory heap usage, each time when the GC
> happend, heap usage will drop down from 6G to 1G, and check the casandra
> log, I found the freeze happend at exactly same times.
>
> So I think when using huge memory(>2G), maybe need using some different GC
> stratege other than the default one provide by Cassandra lunch script.
> Dose't anyone meet this situation, can you please provide some guide?
>
>
> Thanks
> -Santal
>
> 2010/2/17 Tatu Saloranta 
>>
>> On Tue, Feb 16, 2010 at 6:25 AM, Boris Shulman  wrote:
>> > Hello, I'm running some benchmarks on 2 cassandra nodes each running
>> > on 8 cores machine with 16G RAM, 10G for Java heap. I've noticed that
>> > during benchmarks with numerous writes cassandra just freeze for
>> > several minutes (in those benchmarks I'm writing batches of 10 columns
>> > with 1K data each for every key in a single CF). Usually after
>> > performing 50K writes I'm getting a TimeOutException and cassandra
>> > just freezes. What configuration changes can I make in order to
>> > prevent this? Is it possible that my setup just can't handle the load?
>> > How can I calculate the number of casandra nodes for a desired load?
>>
>> One thing that can cause seeming lockups is garbage collector. So
>> enabling GC debug output would be heplful, to see GC activity. Some
>> collector (CMS specifically) can stop the system for very long time,
>> up to minutes. This is not necessarily the root cause, but is easy to
>> rule out.
>> Beyond this, getting a stack trace during lockup would make sense.
>> That can pinpoint what threads are doing, or what they are blocked on
>> in case there is a deadlock or heavy contention on some shared
>> resource.
>>
>> -+ Tatu +-
>
>

Re: cassandra freezes

[Santal Li]

I meet almost same thing as you. When I do some benchmarks write test, some
times one Cassandra will freeze and other node will consider it was shutdown
and up after 30+ second. I am using 5 node, each node 8G mem for java heap.

>From my investigate, it was caused by GC thread, because I start the
JConsole and monitor with the memory heap usage, each time when the GC
happend, heap usage will drop down from 6G to 1G, and check the casandra
log, I found the freeze happend at exactly same times.

So I think when using huge memory(>2G), maybe need using some different GC
stratege other than the default one provide by Cassandra lunch script.
Dose't anyone meet this situation, can you please provide some guide?


Thanks
-Santal

2010/2/17 Tatu Saloranta 

> On Tue, Feb 16, 2010 at 6:25 AM, Boris Shulman  wrote:
> > Hello, I'm running some benchmarks on 2 cassandra nodes each running
> > on 8 cores machine with 16G RAM, 10G for Java heap. I've noticed that
> > during benchmarks with numerous writes cassandra just freeze for
> > several minutes (in those benchmarks I'm writing batches of 10 columns
> > with 1K data each for every key in a single CF). Usually after
> > performing 50K writes I'm getting a TimeOutException and cassandra
> > just freezes. What configuration changes can I make in order to
> > prevent this? Is it possible that my setup just can't handle the load?
> > How can I calculate the number of casandra nodes for a desired load?
>
> One thing that can cause seeming lockups is garbage collector. So
> enabling GC debug output would be heplful, to see GC activity. Some
> collector (CMS specifically) can stop the system for very long time,
> up to minutes. This is not necessarily the root cause, but is easy to
> rule out.
> Beyond this, getting a stack trace during lockup would make sense.
> That can pinpoint what threads are doing, or what they are blocked on
> in case there is a deadlock or heavy contention on some shared
> resource.
>
> -+ Tatu +-
>

Re: Cassandra error with large connection

[Jonathan Ellis]

On Fri, Feb 19, 2010 at 5:55 PM, Adam Fisk  wrote:
> If you decrease the MX value, you'll be
> able to create more threads, but you'll still almost certainly hit the
> same issue eventually.

you can also reduce the per-thread stack size.

but really the right answer is "pool your client connections."

Re: Cassandra error with large connection

[Adam Fisk]

At least on Windows this particular error is quite unique. Oddly,
increasing the maximum heap value *makes it worse.*  The key is the
"native" part of the message -- it's running out of main OS memory
that's getting allocated outside of the JVM.

The overall issue is allocating too many threads. I'm guessing you're
using a cached thread pool. If you decrease the MX value, you'll be
able to create more threads, but you'll still almost certainly hit the
same issue eventually.

-Adam


On Tue, Feb 2, 2010 at 3:06 AM, JKnight JKnight  wrote:
> Thank you very much, Mr Jonathan.
>
> On Mon, Feb 1, 2010 at 11:04 AM, Jonathan Ellis  wrote:
>>
>> On Mon, Feb 1, 2010 at 10:03 AM, Jonathan Ellis  wrote:
>> >> I see a lot of CLOSE_WAIT TCP connection.
>>
>> Also, this sounds like you are not properly pooling client connections
>> to casssandra.  You should have one connection per user, not one
>> connection per operation.
>>
>> -Jonathan
>
>
>
> --
> Best regards,
> JKnight
>



-- 
Adam Fisk
http://www.littleshoot.org | http://adamfisk.wordpress.com |
http://twitter.com/adamfisk

Re: Testing row cache feature in trunk: write should put record in cache

[Jonathan Ellis]

yes.

mmap isn't magic, it's just better than managing that yourself.

On Fri, Feb 19, 2010 at 4:25 PM, Weijun Li  wrote:
> Is it in trunk too? I'm running trunk build (got from the end of last week)
> in cluster and saw the disk i/o bottleneck.
>
> On Fri, Feb 19, 2010 at 1:03 PM, Jonathan Ellis  wrote:
>>
>> mmap is designed to handle that case, yes.  it is already in 0.6 branch.
>>
>> On Fri, Feb 19, 2010 at 2:44 PM, Weijun Li  wrote:
>> > I see. How much is the overhead of java serialization? Does it slow down
>> > the
>> > system a lot? It seems to be a tradeoff between CPU usage and memory.
>> >
>> > As for mmap of 0.6, do you mmap the sstable data file even it is a lot
>> > larger than the available memory (e.g., the data file is over 100GB
>> > while
>> > you have only 8GB ram)? How efficient is mmap in this case? Is mmap
>> > already
>> > checked into 0.6 branch?
>> >
>> > -Weijun
>> >
>> > On Fri, Feb 19, 2010 at 4:56 AM, Jonathan Ellis 
>> > wrote:
>> >>
>> >> The whole point of rowcache is to avoid the serialization overhead,
>> >> though.  If we just wanted the serialized form cached, we would let
>> >> the os block cache handle that without adding an extra layer.  (0.6
>> >> uses mmap'd i/o by default on 64bit JVMs so this is very efficient.)
>> >>
>> >> On Fri, Feb 19, 2010 at 3:29 AM, Weijun Li  wrote:
>> >> > The memory overhead issue is not directly related to GC because when
>> >> > JVM
>> >> > ran
>> >> > out of memory the GC has been very busy for quite a while. In my case
>> >> > JVM
>> >> > consumed all of the 6GB when the row cache size hit 1.4mil.
>> >> >
>> >> > I haven't started test the row cache feature yet. But I think data
>> >> > compression is useful to reduce memory consumption because in my
>> >> > impression
>> >> > disk i/o is always the bottleneck for Cassandra while its CPU usage
>> >> > is
>> >> > usually low all the time. In addition to this, compression should
>> >> > also
>> >> > help
>> >> > to reduce the number of java objects dramatically (correct me if I'm
>> >> > wrong),
>> >> > --especially in case we need to cache most of the data to achieve
>> >> > decent
>> >> > read latency.
>> >> >
>> >> > If ColumnFamily is serializable it shouldn't be that hard to
>> >> > implement
>> >> > the
>> >> > compression feature which can be controlled by an option (again :-)
>> >> > in
>> >> > storage conf xml.
>> >> >
>> >> > When I get to that point you can instruct me to implement this
>> >> > feature
>> >> > along
>> >> > with the row-cache-write-through. Our goal is straightforward: to
>> >> > support
>> >> > short read latency in high volume web application with write/read
>> >> > ratio
>> >> > to
>> >> > be 1:1.
>> >> >
>> >> > -Weijun
>> >> >
>> >> > -Original Message-
>> >> > From: Jonathan Ellis [mailto:jbel...@gmail.com]
>> >> > Sent: Thursday, February 18, 2010 12:04 PM
>> >> > To: cassandra-user@incubator.apache.org
>> >> > Subject: Re: Testing row cache feature in trunk: write should put
>> >> > record
>> >> > in
>> >> > cache
>> >> >
>> >> > Did you force a GC from jconsole to make sure you weren't just
>> >> > measuring uncollected garbage?
>> >> >
>> >> > On Wed, Feb 17, 2010 at 2:51 PM, Weijun Li 
>> >> > wrote:
>> >> >> OK I'll work on the change later because there's another problem to
>> >> >> solve:
>> >> >> the overhead for cache is too big that 1.4mil records (1k each)
>> >> >> consumed
>> >> > all
>> >> >> of the 6gb memory of JVM (I guess 4gb are consumed by the row
>> >> >> cache).
>> >> >> I'm
>> >> >> thinking that ConcurrentHashMap is not a good choice for LRU and the
>> >> >> row
>> >> >> cache needs to store compressed key data to reduce memory usage.
>> >> >> I'll
>> >> >> do
>> >> >> more investigation on this and let you know.
>> >> >>
>> >> >> -Weijun
>> >> >>
>> >> >> On Tue, Feb 16, 2010 at 9:22 PM, Jonathan Ellis 
>> >> >> wrote:
>> >> >>>
>> >> >>> ... tell you what, if you write the option-processing part in
>> >> >>> DatabaseDescriptor I will do the actual cache part. :)
>> >> >>>
>> >> >>> On Tue, Feb 16, 2010 at 11:07 PM, Jonathan Ellis
>> >> >>> 
>> >> >>> wrote:
>> >> >>> > https://issues.apache.org/jira/secure/CreateIssue!default.jspa,
>> >> >>> > but
>> >> >>> > this is pretty low priority for me.
>> >> >>> >
>> >> >>> > On Tue, Feb 16, 2010 at 8:37 PM, Weijun Li 
>> >> >>> > wrote:
>> >> >>> >> Just tried to make quick change to enable it but it didn't work
>> >> >>> >> out
>> >> > :-(
>> >> >>> >>
>> >> >>> >>    ColumnFamily cachedRow =
>> >> >>> >> cfs.getRawCachedRow(mutation.key());
>> >> >>> >>
>> >> >>> >>     // What I modified
>> >> >>> >>     if( cachedRow == null ) {
>> >> >>> >>         cfs.cacheRow(mutation.key());
>> >> >>> >>         cachedRow =
>> >> >>> >> cfs.getRawCachedRow(mutation.key());
>> >> >>> >>     }
>> >> >>> >>
>> >> >>> >>     if (cachedRow != null)
>> >> >>> >>         cachedRow.addAll(columnF

Re: Testing row cache feature in trunk: write should put record in cache

[Weijun Li]

Is it in trunk too? I'm running trunk build (got from the end of last week)
in cluster and saw the disk i/o bottleneck.

On Fri, Feb 19, 2010 at 1:03 PM, Jonathan Ellis  wrote:

> mmap is designed to handle that case, yes.  it is already in 0.6 branch.
>
> On Fri, Feb 19, 2010 at 2:44 PM, Weijun Li  wrote:
> > I see. How much is the overhead of java serialization? Does it slow down
> the
> > system a lot? It seems to be a tradeoff between CPU usage and memory.
> >
> > As for mmap of 0.6, do you mmap the sstable data file even it is a lot
> > larger than the available memory (e.g., the data file is over 100GB while
> > you have only 8GB ram)? How efficient is mmap in this case? Is mmap
> already
> > checked into 0.6 branch?
> >
> > -Weijun
> >
> > On Fri, Feb 19, 2010 at 4:56 AM, Jonathan Ellis 
> wrote:
> >>
> >> The whole point of rowcache is to avoid the serialization overhead,
> >> though.  If we just wanted the serialized form cached, we would let
> >> the os block cache handle that without adding an extra layer.  (0.6
> >> uses mmap'd i/o by default on 64bit JVMs so this is very efficient.)
> >>
> >> On Fri, Feb 19, 2010 at 3:29 AM, Weijun Li  wrote:
> >> > The memory overhead issue is not directly related to GC because when
> JVM
> >> > ran
> >> > out of memory the GC has been very busy for quite a while. In my case
> >> > JVM
> >> > consumed all of the 6GB when the row cache size hit 1.4mil.
> >> >
> >> > I haven't started test the row cache feature yet. But I think data
> >> > compression is useful to reduce memory consumption because in my
> >> > impression
> >> > disk i/o is always the bottleneck for Cassandra while its CPU usage is
> >> > usually low all the time. In addition to this, compression should also
> >> > help
> >> > to reduce the number of java objects dramatically (correct me if I'm
> >> > wrong),
> >> > --especially in case we need to cache most of the data to achieve
> decent
> >> > read latency.
> >> >
> >> > If ColumnFamily is serializable it shouldn't be that hard to implement
> >> > the
> >> > compression feature which can be controlled by an option (again :-) in
> >> > storage conf xml.
> >> >
> >> > When I get to that point you can instruct me to implement this feature
> >> > along
> >> > with the row-cache-write-through. Our goal is straightforward: to
> >> > support
> >> > short read latency in high volume web application with write/read
> ratio
> >> > to
> >> > be 1:1.
> >> >
> >> > -Weijun
> >> >
> >> > -Original Message-
> >> > From: Jonathan Ellis [mailto:jbel...@gmail.com]
> >> > Sent: Thursday, February 18, 2010 12:04 PM
> >> > To: cassandra-user@incubator.apache.org
> >> > Subject: Re: Testing row cache feature in trunk: write should put
> record
> >> > in
> >> > cache
> >> >
> >> > Did you force a GC from jconsole to make sure you weren't just
> >> > measuring uncollected garbage?
> >> >
> >> > On Wed, Feb 17, 2010 at 2:51 PM, Weijun Li 
> wrote:
> >> >> OK I'll work on the change later because there's another problem to
> >> >> solve:
> >> >> the overhead for cache is too big that 1.4mil records (1k each)
> >> >> consumed
> >> > all
> >> >> of the 6gb memory of JVM (I guess 4gb are consumed by the row cache).
> >> >> I'm
> >> >> thinking that ConcurrentHashMap is not a good choice for LRU and the
> >> >> row
> >> >> cache needs to store compressed key data to reduce memory usage. I'll
> >> >> do
> >> >> more investigation on this and let you know.
> >> >>
> >> >> -Weijun
> >> >>
> >> >> On Tue, Feb 16, 2010 at 9:22 PM, Jonathan Ellis 
> >> >> wrote:
> >> >>>
> >> >>> ... tell you what, if you write the option-processing part in
> >> >>> DatabaseDescriptor I will do the actual cache part. :)
> >> >>>
> >> >>> On Tue, Feb 16, 2010 at 11:07 PM, Jonathan Ellis  >
> >> >>> wrote:
> >> >>> > https://issues.apache.org/jira/secure/CreateIssue!default.jspa,
> but
> >> >>> > this is pretty low priority for me.
> >> >>> >
> >> >>> > On Tue, Feb 16, 2010 at 8:37 PM, Weijun Li 
> >> >>> > wrote:
> >> >>> >> Just tried to make quick change to enable it but it didn't work
> out
> >> > :-(
> >> >>> >>
> >> >>> >>ColumnFamily cachedRow =
> >> >>> >> cfs.getRawCachedRow(mutation.key());
> >> >>> >>
> >> >>> >> // What I modified
> >> >>> >> if( cachedRow == null ) {
> >> >>> >> cfs.cacheRow(mutation.key());
> >> >>> >> cachedRow =
> >> >>> >> cfs.getRawCachedRow(mutation.key());
> >> >>> >> }
> >> >>> >>
> >> >>> >> if (cachedRow != null)
> >> >>> >> cachedRow.addAll(columnFamily);
> >> >>> >>
> >> >>> >> How can I open a ticket for you to make the change (enable row
> >> >>> >> cache
> >> >>> >> write
> >> >>> >> through with an option)?
> >> >>> >>
> >> >>> >> Thanks,
> >> >>> >> -Weijun
> >> >>> >>
> >> >>> >> On Tue, Feb 16, 2010 at 5:20 PM, Jonathan Ellis <
> jbel...@

Re: Unbalanced read latency among nodes in a cluster

[Jonathan Ellis]

http://wiki.apache.org/cassandra/Operations

On Fri, Feb 19, 2010 at 3:03 PM, Weijun Li  wrote:
> I setup a two cassandra clusters with 2 nodes each. Both use random
> partitioner. It's strange that for each cluster, one node has much shortter
> read latency than the other one
>
> This is the info of one of the cluster:
>
> Node A: read count 77302, data file 41GB, read latency 58180, io saturation
> 100%
> Node B: read count 488753, data file 26GB, read latency 5822 , io saturation
> 35%.
>
> I first started node A, then ran B to join the cluster. Both machines have
> exactly the same hardware and OS. The test client randomly pick a node to
> write and it worked fine for the other cluster.
>
> Address   Status Load
> Range  Ring
>
> 169400792707028208569145873749456918214
> 10.xxx Up 38.39 GB
> 103633195217832666843316719920043079797    |<--|
> 10.xxx Up 24.22 GB
> 169400792707028208569145873749456918214    |-->|
>
> For both clusters, whichever node that took more reads (with larger data
> file) owns the much worse read latency.
>
> What's the algorithm that cassandra use to split token when a new node is
> joining? What could cause this unbalanced read latency issue? How can I fix
> this? How to make sure all nodes get evenly distributed data and traffic?
>
> -Weijun
>

Re: Testing row cache feature in trunk: write should put record in cache

[Jonathan Ellis]

mmap is designed to handle that case, yes.  it is already in 0.6 branch.

On Fri, Feb 19, 2010 at 2:44 PM, Weijun Li  wrote:
> I see. How much is the overhead of java serialization? Does it slow down the
> system a lot? It seems to be a tradeoff between CPU usage and memory.
>
> As for mmap of 0.6, do you mmap the sstable data file even it is a lot
> larger than the available memory (e.g., the data file is over 100GB while
> you have only 8GB ram)? How efficient is mmap in this case? Is mmap already
> checked into 0.6 branch?
>
> -Weijun
>
> On Fri, Feb 19, 2010 at 4:56 AM, Jonathan Ellis  wrote:
>>
>> The whole point of rowcache is to avoid the serialization overhead,
>> though.  If we just wanted the serialized form cached, we would let
>> the os block cache handle that without adding an extra layer.  (0.6
>> uses mmap'd i/o by default on 64bit JVMs so this is very efficient.)
>>
>> On Fri, Feb 19, 2010 at 3:29 AM, Weijun Li  wrote:
>> > The memory overhead issue is not directly related to GC because when JVM
>> > ran
>> > out of memory the GC has been very busy for quite a while. In my case
>> > JVM
>> > consumed all of the 6GB when the row cache size hit 1.4mil.
>> >
>> > I haven't started test the row cache feature yet. But I think data
>> > compression is useful to reduce memory consumption because in my
>> > impression
>> > disk i/o is always the bottleneck for Cassandra while its CPU usage is
>> > usually low all the time. In addition to this, compression should also
>> > help
>> > to reduce the number of java objects dramatically (correct me if I'm
>> > wrong),
>> > --especially in case we need to cache most of the data to achieve decent
>> > read latency.
>> >
>> > If ColumnFamily is serializable it shouldn't be that hard to implement
>> > the
>> > compression feature which can be controlled by an option (again :-) in
>> > storage conf xml.
>> >
>> > When I get to that point you can instruct me to implement this feature
>> > along
>> > with the row-cache-write-through. Our goal is straightforward: to
>> > support
>> > short read latency in high volume web application with write/read ratio
>> > to
>> > be 1:1.
>> >
>> > -Weijun
>> >
>> > -Original Message-
>> > From: Jonathan Ellis [mailto:jbel...@gmail.com]
>> > Sent: Thursday, February 18, 2010 12:04 PM
>> > To: cassandra-user@incubator.apache.org
>> > Subject: Re: Testing row cache feature in trunk: write should put record
>> > in
>> > cache
>> >
>> > Did you force a GC from jconsole to make sure you weren't just
>> > measuring uncollected garbage?
>> >
>> > On Wed, Feb 17, 2010 at 2:51 PM, Weijun Li  wrote:
>> >> OK I'll work on the change later because there's another problem to
>> >> solve:
>> >> the overhead for cache is too big that 1.4mil records (1k each)
>> >> consumed
>> > all
>> >> of the 6gb memory of JVM (I guess 4gb are consumed by the row cache).
>> >> I'm
>> >> thinking that ConcurrentHashMap is not a good choice for LRU and the
>> >> row
>> >> cache needs to store compressed key data to reduce memory usage. I'll
>> >> do
>> >> more investigation on this and let you know.
>> >>
>> >> -Weijun
>> >>
>> >> On Tue, Feb 16, 2010 at 9:22 PM, Jonathan Ellis 
>> >> wrote:
>> >>>
>> >>> ... tell you what, if you write the option-processing part in
>> >>> DatabaseDescriptor I will do the actual cache part. :)
>> >>>
>> >>> On Tue, Feb 16, 2010 at 11:07 PM, Jonathan Ellis 
>> >>> wrote:
>> >>> > https://issues.apache.org/jira/secure/CreateIssue!default.jspa, but
>> >>> > this is pretty low priority for me.
>> >>> >
>> >>> > On Tue, Feb 16, 2010 at 8:37 PM, Weijun Li 
>> >>> > wrote:
>> >>> >> Just tried to make quick change to enable it but it didn't work out
>> > :-(
>> >>> >>
>> >>> >>    ColumnFamily cachedRow =
>> >>> >> cfs.getRawCachedRow(mutation.key());
>> >>> >>
>> >>> >>     // What I modified
>> >>> >>     if( cachedRow == null ) {
>> >>> >>         cfs.cacheRow(mutation.key());
>> >>> >>         cachedRow =
>> >>> >> cfs.getRawCachedRow(mutation.key());
>> >>> >>     }
>> >>> >>
>> >>> >>     if (cachedRow != null)
>> >>> >>         cachedRow.addAll(columnFamily);
>> >>> >>
>> >>> >> How can I open a ticket for you to make the change (enable row
>> >>> >> cache
>> >>> >> write
>> >>> >> through with an option)?
>> >>> >>
>> >>> >> Thanks,
>> >>> >> -Weijun
>> >>> >>
>> >>> >> On Tue, Feb 16, 2010 at 5:20 PM, Jonathan Ellis 
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> On Tue, Feb 16, 2010 at 7:17 PM, Jonathan Ellis
>> >>> >>> 
>> >>> >>> wrote:
>> >>> >>> > On Tue, Feb 16, 2010 at 7:11 PM, Weijun Li 
>> >>> >>> > wrote:
>> >>> >>> >> Just started to play with the row cache feature in trunk: it
>> >>> >>> >> seems
>> >>> >>> >> to
>> >>> >>> >> be
>> >>> >>> >> working fine so far except that for RowsCached parameter you
>> >>> >>> >> need
>> >>> >>> >> to
>> >>> >>> >> specify
>> >>> >>> >> number of rows rather than a percentage (e

Unbalanced read latency among nodes in a cluster

[Weijun Li]

I setup a two cassandra clusters with 2 nodes each. Both use random
partitioner. It's strange that for each cluster, one node has much shortter
read latency than the other one

This is the info of one of the cluster:

Node A: read count 77302, data file 41GB, read latency 58180, io saturation
100%
Node B: read count 488753, data file 26GB, read latency 5822 , io saturation
35%.

I first started node A, then ran B to join the cluster. Both machines have
exactly the same hardware and OS. The test client randomly pick a node to
write and it worked fine for the other cluster.

Address   Status Load
Range  Ring

169400792707028208569145873749456918214
10.xxx Up 38.39 GB
103633195217832666843316719920043079797|<--|
10.xxx Up 24.22 GB
169400792707028208569145873749456918214|-->|

For both clusters, whichever node that took more reads (with larger data
file) owns the much worse read latency.

What's the algorithm that cassandra use to split token when a new node is
joining? What could cause this unbalanced read latency issue? How can I fix
this? How to make sure all nodes get evenly distributed data and traffic?

-Weijun

Re: Testing row cache feature in trunk: write should put record in cache

[Weijun Li]

I see. How much is the overhead of java serialization? Does it slow down the
system a lot? It seems to be a tradeoff between CPU usage and memory.

As for mmap of 0.6, do you mmap the sstable data file even it is a lot
larger than the available memory (e.g., the data file is over 100GB while
you have only 8GB ram)? How efficient is mmap in this case? Is mmap already
checked into 0.6 branch?

-Weijun

On Fri, Feb 19, 2010 at 4:56 AM, Jonathan Ellis  wrote:

> The whole point of rowcache is to avoid the serialization overhead,
> though.  If we just wanted the serialized form cached, we would let
> the os block cache handle that without adding an extra layer.  (0.6
> uses mmap'd i/o by default on 64bit JVMs so this is very efficient.)
>
> On Fri, Feb 19, 2010 at 3:29 AM, Weijun Li  wrote:
> > The memory overhead issue is not directly related to GC because when JVM
> ran
> > out of memory the GC has been very busy for quite a while. In my case JVM
> > consumed all of the 6GB when the row cache size hit 1.4mil.
> >
> > I haven't started test the row cache feature yet. But I think data
> > compression is useful to reduce memory consumption because in my
> impression
> > disk i/o is always the bottleneck for Cassandra while its CPU usage is
> > usually low all the time. In addition to this, compression should also
> help
> > to reduce the number of java objects dramatically (correct me if I'm
> wrong),
> > --especially in case we need to cache most of the data to achieve decent
> > read latency.
> >
> > If ColumnFamily is serializable it shouldn't be that hard to implement
> the
> > compression feature which can be controlled by an option (again :-) in
> > storage conf xml.
> >
> > When I get to that point you can instruct me to implement this feature
> along
> > with the row-cache-write-through. Our goal is straightforward: to support
> > short read latency in high volume web application with write/read ratio
> to
> > be 1:1.
> >
> > -Weijun
> >
> > -Original Message-
> > From: Jonathan Ellis [mailto:jbel...@gmail.com]
> > Sent: Thursday, February 18, 2010 12:04 PM
> > To: cassandra-user@incubator.apache.org
> > Subject: Re: Testing row cache feature in trunk: write should put record
> in
> > cache
> >
> > Did you force a GC from jconsole to make sure you weren't just
> > measuring uncollected garbage?
> >
> > On Wed, Feb 17, 2010 at 2:51 PM, Weijun Li  wrote:
> >> OK I'll work on the change later because there's another problem to
> solve:
> >> the overhead for cache is too big that 1.4mil records (1k each) consumed
> > all
> >> of the 6gb memory of JVM (I guess 4gb are consumed by the row cache).
> I'm
> >> thinking that ConcurrentHashMap is not a good choice for LRU and the row
> >> cache needs to store compressed key data to reduce memory usage. I'll do
> >> more investigation on this and let you know.
> >>
> >> -Weijun
> >>
> >> On Tue, Feb 16, 2010 at 9:22 PM, Jonathan Ellis 
> wrote:
> >>>
> >>> ... tell you what, if you write the option-processing part in
> >>> DatabaseDescriptor I will do the actual cache part. :)
> >>>
> >>> On Tue, Feb 16, 2010 at 11:07 PM, Jonathan Ellis 
> >>> wrote:
> >>> > https://issues.apache.org/jira/secure/CreateIssue!default.jspa,
> but
> >>> > this is pretty low priority for me.
> >>> >
> >>> > On Tue, Feb 16, 2010 at 8:37 PM, Weijun Li 
> wrote:
> >>> >> Just tried to make quick change to enable it but it didn't work out
> > :-(
> >>> >>
> >>> >>ColumnFamily cachedRow =
> >>> >> cfs.getRawCachedRow(mutation.key());
> >>> >>
> >>> >> // What I modified
> >>> >> if( cachedRow == null ) {
> >>> >> cfs.cacheRow(mutation.key());
> >>> >> cachedRow = cfs.getRawCachedRow(mutation.key());
> >>> >> }
> >>> >>
> >>> >> if (cachedRow != null)
> >>> >> cachedRow.addAll(columnFamily);
> >>> >>
> >>> >> How can I open a ticket for you to make the change (enable row cache
> >>> >> write
> >>> >> through with an option)?
> >>> >>
> >>> >> Thanks,
> >>> >> -Weijun
> >>> >>
> >>> >> On Tue, Feb 16, 2010 at 5:20 PM, Jonathan Ellis 
> >>> >> wrote:
> >>> >>>
> >>> >>> On Tue, Feb 16, 2010 at 7:17 PM, Jonathan Ellis  >
> >>> >>> wrote:
> >>> >>> > On Tue, Feb 16, 2010 at 7:11 PM, Weijun Li 
> >>> >>> > wrote:
> >>> >>> >> Just started to play with the row cache feature in trunk: it
> seems
> >>> >>> >> to
> >>> >>> >> be
> >>> >>> >> working fine so far except that for RowsCached parameter you
> need
> >>> >>> >> to
> >>> >>> >> specify
> >>> >>> >> number of rows rather than a percentage (e.g., "20%" doesn't
> > work).
> >>> >>> >
> >>> >>> > 20% works, but it's 20% of the rows at server startup.  So on a
> >>> >>> > fresh
> >>> >>> > start that is zero.
> >>> >>> >
> >>> >>> > Maybe we should just get rid of the % feature...
> >>> >>>
> >>> >>> (Actually, it shouldn't be hard to upd

Re: Row with many columns

[Ted Zlatanov]

On Fri, 19 Feb 2010 11:46:25 +0100 Wojciech Kaczmarek  
wrote: 

WK> You can use get_range_slice with the start_key and finish_key crafted like
WK> this:
WK> assume id is "<10-digits-worldId:whatever_objectId>"
WK> to get objects from world with id=42 you do:
WK> start_key='42:', end_key='42~'
WK> (for the ending mark of end_key choose any character which is greater than
WK> your delimiter ':' in a sort order)
WK> Just remember the keys ordering is defined by the specific Partitioner class
WK> used)

It's still not clear what's in each object supercolumn in your current
schema.  How do you encode a relationship between two objects?  How do
you encode an object's properties?  That's the heart of your design.

For the row keys I agree with Wojciech, you need to let Cassandra handle
objects as rows to get good results.

Ted

Re: Cassandra News Page

[Ian Holsman]

Hi Sal.
we'll be moving off the incubator site shortly. we'll address that when we go 
to cassandra.apache.org

regards
Ian
On Feb 18, 2010, at 4:06 PM, Sal Fuentes wrote:

> This is just a thought, but I think some type of *latest news* page would be 
> nice to have on the main site (http://incubator.apache.org/cassandra/) even 
> if its a bit outdated. Not sure if this has been previously considered. 
> 
> -- 
> Salvador Fuentes Jr.

--
Ian Holsman
i...@holsman.net

Re: Testing row cache feature in trunk: write should put record in cache

[Jonathan Ellis]

The whole point of rowcache is to avoid the serialization overhead,
though.  If we just wanted the serialized form cached, we would let
the os block cache handle that without adding an extra layer.  (0.6
uses mmap'd i/o by default on 64bit JVMs so this is very efficient.)

On Fri, Feb 19, 2010 at 3:29 AM, Weijun Li  wrote:
> The memory overhead issue is not directly related to GC because when JVM ran
> out of memory the GC has been very busy for quite a while. In my case JVM
> consumed all of the 6GB when the row cache size hit 1.4mil.
>
> I haven't started test the row cache feature yet. But I think data
> compression is useful to reduce memory consumption because in my impression
> disk i/o is always the bottleneck for Cassandra while its CPU usage is
> usually low all the time. In addition to this, compression should also help
> to reduce the number of java objects dramatically (correct me if I'm wrong),
> --especially in case we need to cache most of the data to achieve decent
> read latency.
>
> If ColumnFamily is serializable it shouldn't be that hard to implement the
> compression feature which can be controlled by an option (again :-) in
> storage conf xml.
>
> When I get to that point you can instruct me to implement this feature along
> with the row-cache-write-through. Our goal is straightforward: to support
> short read latency in high volume web application with write/read ratio to
> be 1:1.
>
> -Weijun
>
> -Original Message-
> From: Jonathan Ellis [mailto:jbel...@gmail.com]
> Sent: Thursday, February 18, 2010 12:04 PM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Testing row cache feature in trunk: write should put record in
> cache
>
> Did you force a GC from jconsole to make sure you weren't just
> measuring uncollected garbage?
>
> On Wed, Feb 17, 2010 at 2:51 PM, Weijun Li  wrote:
>> OK I'll work on the change later because there's another problem to solve:
>> the overhead for cache is too big that 1.4mil records (1k each) consumed
> all
>> of the 6gb memory of JVM (I guess 4gb are consumed by the row cache). I'm
>> thinking that ConcurrentHashMap is not a good choice for LRU and the row
>> cache needs to store compressed key data to reduce memory usage. I'll do
>> more investigation on this and let you know.
>>
>> -Weijun
>>
>> On Tue, Feb 16, 2010 at 9:22 PM, Jonathan Ellis  wrote:
>>>
>>> ... tell you what, if you write the option-processing part in
>>> DatabaseDescriptor I will do the actual cache part. :)
>>>
>>> On Tue, Feb 16, 2010 at 11:07 PM, Jonathan Ellis 
>>> wrote:
>>> > https://issues.apache.org/jira/secure/CreateIssue!default.jspa, but
>>> > this is pretty low priority for me.
>>> >
>>> > On Tue, Feb 16, 2010 at 8:37 PM, Weijun Li  wrote:
>>> >> Just tried to make quick change to enable it but it didn't work out
> :-(
>>> >>
>>> >>    ColumnFamily cachedRow =
>>> >> cfs.getRawCachedRow(mutation.key());
>>> >>
>>> >>     // What I modified
>>> >>     if( cachedRow == null ) {
>>> >>         cfs.cacheRow(mutation.key());
>>> >>         cachedRow = cfs.getRawCachedRow(mutation.key());
>>> >>     }
>>> >>
>>> >>     if (cachedRow != null)
>>> >>         cachedRow.addAll(columnFamily);
>>> >>
>>> >> How can I open a ticket for you to make the change (enable row cache
>>> >> write
>>> >> through with an option)?
>>> >>
>>> >> Thanks,
>>> >> -Weijun
>>> >>
>>> >> On Tue, Feb 16, 2010 at 5:20 PM, Jonathan Ellis 
>>> >> wrote:
>>> >>>
>>> >>> On Tue, Feb 16, 2010 at 7:17 PM, Jonathan Ellis 
>>> >>> wrote:
>>> >>> > On Tue, Feb 16, 2010 at 7:11 PM, Weijun Li 
>>> >>> > wrote:
>>> >>> >> Just started to play with the row cache feature in trunk: it seems
>>> >>> >> to
>>> >>> >> be
>>> >>> >> working fine so far except that for RowsCached parameter you need
>>> >>> >> to
>>> >>> >> specify
>>> >>> >> number of rows rather than a percentage (e.g., "20%" doesn't
> work).
>>> >>> >
>>> >>> > 20% works, but it's 20% of the rows at server startup.  So on a
>>> >>> > fresh
>>> >>> > start that is zero.
>>> >>> >
>>> >>> > Maybe we should just get rid of the % feature...
>>> >>>
>>> >>> (Actually, it shouldn't be hard to update this on flush, if you want
>>> >>> to open a ticket.)
>>> >>
>>> >>
>>> >
>>
>>
>
>

Re: Row with many columns

[Wojciech Kaczmarek]

You can use get_range_slice with the start_key and finish_key crafted like
this:
assume id is "<10-digits-worldId:whatever_objectId>"
to get objects from world with id=42 you do:
start_key='42:', end_key='42~'
(for the ending mark of end_key choose any character which is greater than
your delimiter ':' in a sort order)
Just remember the keys ordering is defined by the specific Partitioner class
used)

On Fri, Feb 19, 2010 at 08:40, ruslan usifov wrote:

> But how can i do range query?, for example get all objects of wolrd?
>
> 2010/2/19 Wojciech Kaczmarek 
>
> Hi Ruslan,
>>
>> I'd model your objects as rows. You can have a concatenation of worldId &
>> objectId as a key, or just objectId alone.
>>
>>
>> On Thu, Feb 18, 2010 at 23:18, ruslan usifov wrote:
>>
>>> For example i have worlds with objects in it (stones, trees and so on).
>>> In my case worldid is key, objects are supercollums(some extra info or draw)
>>>
>>>
>>> 2010/2/18 Ted Zlatanov 
>>>
 On Thu, 18 Feb 2010 18:05:50 +0300 ruslan usifov <
 ruslan.usi...@gmail.com> wrote:

 ru> 2010/2/18 Ted Zlatanov 
 >> Maybe you should partition your data differently?  What does each of
 >> these tiny columns represent and how do you allocate the row key?

 ru> Friends relations, for statistic

 Sorry, that's not enough information.  Can you give an example of some
 columns and what they mean, and also some row keys and what they mean?

 Ted


>>>
>>
>

RE: Testing row cache feature in trunk: write should put record in cache

[Weijun Li]

The memory overhead issue is not directly related to GC because when JVM ran
out of memory the GC has been very busy for quite a while. In my case JVM
consumed all of the 6GB when the row cache size hit 1.4mil.

I haven't started test the row cache feature yet. But I think data
compression is useful to reduce memory consumption because in my impression
disk i/o is always the bottleneck for Cassandra while its CPU usage is
usually low all the time. In addition to this, compression should also help
to reduce the number of java objects dramatically (correct me if I'm wrong),
--especially in case we need to cache most of the data to achieve decent
read latency.

If ColumnFamily is serializable it shouldn't be that hard to implement the
compression feature which can be controlled by an option (again :-) in
storage conf xml.

When I get to that point you can instruct me to implement this feature along
with the row-cache-write-through. Our goal is straightforward: to support
short read latency in high volume web application with write/read ratio to
be 1:1.

-Weijun

-Original Message-
From: Jonathan Ellis [mailto:jbel...@gmail.com] 
Sent: Thursday, February 18, 2010 12:04 PM
To: cassandra-user@incubator.apache.org
Subject: Re: Testing row cache feature in trunk: write should put record in
cache

Did you force a GC from jconsole to make sure you weren't just
measuring uncollected garbage?

On Wed, Feb 17, 2010 at 2:51 PM, Weijun Li  wrote:
> OK I'll work on the change later because there's another problem to solve:
> the overhead for cache is too big that 1.4mil records (1k each) consumed
all
> of the 6gb memory of JVM (I guess 4gb are consumed by the row cache). I'm
> thinking that ConcurrentHashMap is not a good choice for LRU and the row
> cache needs to store compressed key data to reduce memory usage. I'll do
> more investigation on this and let you know.
>
> -Weijun
>
> On Tue, Feb 16, 2010 at 9:22 PM, Jonathan Ellis  wrote:
>>
>> ... tell you what, if you write the option-processing part in
>> DatabaseDescriptor I will do the actual cache part. :)
>>
>> On Tue, Feb 16, 2010 at 11:07 PM, Jonathan Ellis 
>> wrote:
>> > https://issues.apache.org/jira/secure/CreateIssue!default.jspa, but
>> > this is pretty low priority for me.
>> >
>> > On Tue, Feb 16, 2010 at 8:37 PM, Weijun Li  wrote:
>> >> Just tried to make quick change to enable it but it didn't work out
:-(
>> >>
>> >>    ColumnFamily cachedRow =
>> >> cfs.getRawCachedRow(mutation.key());
>> >>
>> >>     // What I modified
>> >>     if( cachedRow == null ) {
>> >>         cfs.cacheRow(mutation.key());
>> >>         cachedRow = cfs.getRawCachedRow(mutation.key());
>> >>     }
>> >>
>> >>     if (cachedRow != null)
>> >>         cachedRow.addAll(columnFamily);
>> >>
>> >> How can I open a ticket for you to make the change (enable row cache
>> >> write
>> >> through with an option)?
>> >>
>> >> Thanks,
>> >> -Weijun
>> >>
>> >> On Tue, Feb 16, 2010 at 5:20 PM, Jonathan Ellis 
>> >> wrote:
>> >>>
>> >>> On Tue, Feb 16, 2010 at 7:17 PM, Jonathan Ellis 
>> >>> wrote:
>> >>> > On Tue, Feb 16, 2010 at 7:11 PM, Weijun Li 
>> >>> > wrote:
>> >>> >> Just started to play with the row cache feature in trunk: it seems
>> >>> >> to
>> >>> >> be
>> >>> >> working fine so far except that for RowsCached parameter you need
>> >>> >> to
>> >>> >> specify
>> >>> >> number of rows rather than a percentage (e.g., "20%" doesn't
work).
>> >>> >
>> >>> > 20% works, but it's 20% of the rows at server startup.  So on a
>> >>> > fresh
>> >>> > start that is zero.
>> >>> >
>> >>> > Maybe we should just get rid of the % feature...
>> >>>
>> >>> (Actually, it shouldn't be hard to update this on flush, if you want
>> >>> to open a ticket.)
>> >>
>> >>
>> >
>
>