One thing I have seen in the past when benchmarking is that the client GC settings are equally important. When the client GC pauses, to collect all those small short lived objects, it can make the server look bad.
I'd use similar GC settings (with a smaller heap) then I'd use on the server (e.g. parallel GC). On Mon, Dec 14, 2009 at 6:03 PM, Brian Burruss <bburr...@real.com> wrote: > I plan to investigate that next. There isn't a straightforward "disconnect" > that I could easily see. > > Ian Holsman <i...@holsman.net> wrote: > > > can you make it so that the client restarts the connection every 30m or so ? > It could be an issue in thrift or something with long-lived connections. > > On Dec 15, 2009, at 10:16 AM, Brian Burruss wrote: > >> i agree. i don't know anything about thrift, and i don't know how it keeps >> connections open or manages resources from a client or server perspective, >> but this situation suggests that maybe killing the clients is forcing the >> server to free something. >> >> how's that sound :) >> >> ________________________________________ >> From: Jonathan Ellis [jbel...@gmail.com] >> Sent: Monday, December 14, 2009 3:12 PM >> To: cassandra-user@incubator.apache.org >> Subject: Re: read latency creaping up >> >> hmm, me neither >> >> but, I can't think how restarting the client would, either :) >> >> On Mon, Dec 14, 2009 at 4:59 PM, Brian Burruss <bburr...@real.com> wrote: >>> Well not sure how that would affect he latency as reported by the Cassandra >>> server using nodeprobe cfstats >>> >>> Jonathan Ellis <jbel...@gmail.com> wrote: >>> >>> >>> possibly the clients are running into memory pressure? >>> >>> On Mon, Dec 14, 2009 at 4:27 PM, Brian Burruss <bburr...@real.com> wrote: >>>> thx, i'm actually the "B. Todd Burruss" in that thread .. we changed our >>>> email system and well now, i'm just Brian .. long story. >>>> >>>> anyway, in this case it isn't compaction pendings as i can kill the >>>> clients and immediately restart and the latency is back to a reasonable >>>> number. i'm still investigating. >>>> >>>> thx! >>>> ________________________________________ >>>> From: Eric Evans [eev...@rackspace.com] >>>> Sent: Monday, December 14, 2009 8:23 AM >>>> To: cassandra-user@incubator.apache.org >>>> Subject: RE: read latency creaping up >>>> >>>> On Sun, 2009-12-13 at 13:18 -0800, Brian Burruss wrote: >>>>> if this isn't a known issue, lemme do some more investigating. my >>>>> test client becomes "more random" with reads as time progresses, so >>>>> possibly this is what causes the latency issue. however, all that >>>>> being said, the performance really becomes bad after a while. >>>> >>>> Have a look at the following thread: >>>> >>>> http://thread.gmane.org/gmane.comp.db.cassandra.user/1402 >>>> >>>> >>>> -- >>>> Eric Evans >>>> eev...@rackspace.com >>>> >>>> >>> > > -- > Ian Holsman > i...@holsman.net > > > >
