Rohit, Would you please run “riak-debug” on one server from the command line and send the tar.gz file it creates directly to me. Do not copy the email list.
Notes on its usage are here: http://docs.basho.com/riak/kv/2.2.1/using/cluster-operations/inspecting-node/#riak-debug <http://docs.basho.com/riak/kv/2.2.1/using/cluster-operations/inspecting-node/#riak-debug> The resulting debug package will give me and others at Basho a better picture of the problem. The alternative is a about twenty rounds of “what about this, oh, then what about that”. Thanks, Matthew > On Mar 21, 2017, at 9:53 PM, Rohit Sanbhadti <sanbhadtiro...@vmware.com> > wrote: > > Matthew, > > To clarify, this happens on all nodes in our cluster (10+ nodes) although the > exact size varies by 10s of GB. I’ve rolling restarted the cluster the last > time this happened (last week) with no significant change in size, although > the output from riak-admin aae-status and riak-admin search aae-status shows > empty after the restart. > > -- > Rohit S. > > On 3/21/17, 5:25 PM, "Matthew Von-Maszewski" <matth...@basho.com> wrote: > > Rohit, > > If you restart the node does the elevated anti_entropy size decline after > the restart? > > Matthew > > >> On Mar 21, 2017, at 8:00 PM, Rohit Sanbhadti <sanbhadtiro...@vmware.com> >> wrote: >> >> Running Riak 2.2.0 on Ubuntu 16.04.1, we’ve noticed that anti_entropy is >> taking up way too much space on all of our nodes. We use multi_backend with >> mostly bitcask backends (relevant part of config pasted below). Has anyone >> seen this before, or does anyone have an idea what might be causing this? >> >> storage_backend = multi >> multi_backend.bitcask_1day.storage_backend = bitcask >> multi_backend.bitcask_1day.bitcask.data_root = /var/lib/riak/bitcask_1day >> multi_backend.bitcask_1day.bitcask.expiry = 1d >> multi_backend.bitcask_1day.bitcask.expiry.grace_time = 1h >> multi_backend.bitcask_2day.storage_backend = bitcask >> multi_backend.bitcask_2day.bitcask.data_root = /var/lib/riak/bitcask_2day >> multi_backend.bitcask_2day.bitcask.expiry = 2d >> multi_backend.bitcask_2day.bitcask.expiry.grace_time = 1h >> multi_backend.bitcask_4day.storage_backend = bitcask >> multi_backend.bitcask_4day.bitcask.data_root = /var/lib/riak/bitcask_4day >> multi_backend.bitcask_4day.bitcask.expiry = 4d >> multi_backend.bitcask_4day.bitcask.expiry.grace_time = 1h >> multi_backend.bitcask_8day.storage_backend = bitcask >> multi_backend.bitcask_8day.bitcask.data_root = /var/lib/riak/bitcask_8day >> multi_backend.bitcask_8day.bitcask.expiry = 8d >> multi_backend.bitcask_8day.bitcask.expiry.grace_time = 1h >> multi_backend.leveldb_mult.storage_backend = leveldb >> multi_backend.leveldb_mult.leveldb.maximum_memory.percent = 30 >> multi_backend.leveldb_mult.leveldb.data_root = /var/lib/riak/leveldb_mult >> multi_backend.bitcask_mult.storage_backend = bitcask >> multi_backend.bitcask_mult.bitcask.data_root = /var/lib/riak/bitcask_mult >> multi_backend.default = leveldb_mult >> >> $ du –h –d 1 /var/lib/riak >> >> 4.0K /var/lib/riak/riak_kv_exchange_fsm >> 52K /var/lib/riak/generated.configs >> 224K /var/lib/riak/cluster_meta >> 424K /var/lib/riak/ring >> 1.1M /var/lib/riak/kv_vnode >> 2.1M /var/lib/riak/bitcask_1day >> 3.6M /var/lib/riak/bitcask_4day >> 33M /var/lib/riak/yz_temp >> 1.1G /var/lib/riak/bitcask_2day >> 5.9G /var/lib/riak/yz_anti_entropy >> 20G /var/lib/riak/yz >> 24G /var/lib/riak/leveldb_mult >> 25G /var/lib/riak/bitcask_mult >> 27G /var/lib/riak/bitcask_8day >> 139G /var/lib/riak/anti_entropy >> 240G /var/lib/riak >> >> >> -- >> Rohit S. >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.basho.com_mailman_listinfo_riak-2Dusers-5Flists.basho.com&d=DwIFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=qO7g3ZLnCDJ2Kp8qz13HpzC2NAX5qNCtEhoWeGshuEI&m=zcfqvyFJVK17vAft30Cv9abkK5P_f2JtRonrl0CRYnM&s=wSNoX28DcQ-CCfWp9vRUA-70z3LeWFNGHG3-LPKzAtQ&e= >> > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
