Because it's important to note, the version of Riak is 2.0.5. AAE was actually the first thing I checked - it was on, but turning it off had no impact.
Interestingly enough, I was able to pinpoint the source of the higher than expected CPU to a service that was frequently polling a set of map and set CRDTs from Riak once every second (which does not really seem all that frequent) to keep up with any changes that may have occurred since the last time it cached the information in process. I'm trying to get some time to look more into if we're doing something dumb in our service (always possible) or if there are settings for these CRDTs that we're not making proper use of in Riak itself. For now, I've tuned down the polling frequency for the test environment (which for some reason was higher than the production environment), and the problem appears to have gone away. We didn't seem to see any noticeable rise or fall in the CPU utilization of the service itself, just in Riak If anyone has any insights into poor performance with the map / set CRDT types (maps and sets, and maps containing sets), any input would be welcome to help hunt down why that seemed to be such a issue. On Thu, Aug 6, 2015 at 8:19 AM, Matthew Von-Maszewski <matth...@basho.com> wrote: > Two ideas about lots of cpu with nothing happening: > > - is Riak entropy setting "active". Weekly it will rebuild hash trees > > - does this activity occur after a handoff? Sometimes leveldb needs time > to catch up afterward in current releases (fix coming soon). > > Matthew > > Sent from my iPad > > > On Aug 5, 2015, at 7:16 PM, Matthew Brender <mbren...@basho.com> wrote: > > > > Hey all, I'm practicing what I preach and asking a question to others > > here. A community member pinged me with a couple questions: > > > > "I'm looking into an issue with Riak on our test environment (we > > basically let devs spin up a copy of our prod stack on a giant box in > > ec2). > > > > We've gotten some complaints that riak seems to be using a lot of CPU, > > even when "nothing" is happening (something is always happening with > > the # of apps and services on the box, even when no one is poking it). > > > > Is there a way to turn on an "access" log, so I could see what > > requests are being made against riak, to try and correlate with? It > > would have to be able to tell me about pbuffs usage, not just http." > > > > Could anyone help with these questions. I see a couple separate ones: > > > > 1. How to monitor what is causing higher CPU usage on a Riak cluster > > 2. How to view an access log of protobuff and http requests, or just > > the *most helpful* counters for them > > > > I know a ton of statistics are visible [1] through riak-admin, but I'd > > like to hear what others recommend in particular. > > > > [1] http://docs.basho.com/riak/latest/ops/running/stats-and-monitoring/ > > > > Thanks, > > > > Matt Brender | Developer Advocacy Lead > > Basho Technologies > > t: @mjbrender > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
