On Mon, Apr 3, 2017, at 22:20, Matthew Von-Maszewski wrote: > Rohit, > > My apologies for the delayed reply. Too many conflicting demands on > my time the past two weeks. > > I reviewed the riak-debug package you shared. I also discussed its > contents with other Riak developers. > > There does not appear to be anything unexpected. The anti_entropy > bloat is due to the bitcask backend not actively communicating TTL > expirations to AAE. This is a known issue. Similarly, bitcask is not > communicating TTL expirations to solr. (The leveldb backend recently > added expiry / TTL. And it fails the same way in this scenario as > bitcask.) > > We have engineering designs in the works that will eventually correct > this situation. But the designs do not translate to code that you can > use today. My apologies. Hi Matthew, I've been experiencing the same kind of issues on some application prototype I'm working on. I tried using leveldb backend (through multi- backend) with TTL, coupled to Riak Search, and indeed Solr data are not deleted. Do you know if there is any ticket to follow ? (I searched for a github issue but couldn't find any), or any rough ETA on a fix on this ? Also, are there any best practises about deleting data from KV and Solr, especially, is search.queue.batch.minimum taken in account when deleting keys ? Thanks, Damien > > --> The 100% accurate approach today is to disable bitcask's TTL and > create external jobs that prune your data via Delete operations. Yes, > this is going to create a bunch of extra disk operations. But I am > being honest. > > --> You could reduce only the anti_entropy disk usage by changing the > "anti_entropy" setting in riak.conf from "active" to "passive". But > this does nothing for solr. > > Matthew > >> On Mar 22, 2017, at 10:56 AM, Matthew Von-Maszewski >> <matth...@basho.com> wrote: >> >> Rohit, >> >> Would you please run “riak-debug” on one server from the command line >> and send the tar.gz file it creates directly to me. Do not copy the >> email list. >> >> Notes on its usage are here: >> http://docs.basho.com/riak/kv/2.2.1/using/cluster-operations/inspecting-node/#riak-debug >> >> The resulting debug package will give me and others at Basho a better >> picture of the problem. The alternative is a about twenty rounds of >> “what about this, oh, then what about that”. >> >> Thanks, >> Matthew >> >> >> >>> On Mar 21, 2017, at 9:53 PM, Rohit Sanbhadti >>> <sanbhadtiro...@vmware.com> wrote: >>> >>> Matthew, >>> >>> To clarify, this happens on all nodes in our cluster (10+ nodes) >>> although the exact size varies by 10s of GB. I’ve rolling restarted >>> the cluster the last time this happened (last week) with no >>> significant change in size, although the output from riak-admin aae- >>> status and riak-admin search aae-status shows empty after the >>> restart. >>> >>> -- >>> Rohit S. >>> >>> On 3/21/17, 5:25 PM, "Matthew Von-Maszewski" <matth...@basho.com> >>> wrote: >>> >>> Rohit, >>> >>> If you restart the node does the elevated anti_entropy size >>> decline after the restart? >>> >>> Matthew >>> >>> >>>> On Mar 21, 2017, at 8:00 PM, Rohit Sanbhadti >>>> <sanbhadtiro...@vmware.com> wrote: >>>> >>>> Running Riak 2.2.0 on Ubuntu 16.04.1, we’ve noticed that >>>> anti_entropy is taking up way too much space on all of our nodes. >>>> We use multi_backend with mostly bitcask backends (relevant part of >>>> config pasted below). Has anyone seen this before, or does anyone >>>> have an idea what might be causing this? >>>> >>>> storage_backend = multi >>>> multi_backend.bitcask_1day.storage_backend = bitcask >>>> multi_backend.bitcask_1day.bitcask.data_root = >>>> /var/lib/riak/bitcask_1day >>>> multi_backend.bitcask_1day.bitcask.expiry = 1d >>>> multi_backend.bitcask_1day.bitcask.expiry.grace_time = 1h >>>> multi_backend.bitcask_2day.storage_backend = bitcask >>>> multi_backend.bitcask_2day.bitcask.data_root = >>>> /var/lib/riak/bitcask_2day >>>> multi_backend.bitcask_2day.bitcask.expiry = 2d >>>> multi_backend.bitcask_2day.bitcask.expiry.grace_time = 1h >>>> multi_backend.bitcask_4day.storage_backend = bitcask >>>> multi_backend.bitcask_4day.bitcask.data_root = >>>> /var/lib/riak/bitcask_4day >>>> multi_backend.bitcask_4day.bitcask.expiry = 4d >>>> multi_backend.bitcask_4day.bitcask.expiry.grace_time = 1h >>>> multi_backend.bitcask_8day.storage_backend = bitcask >>>> multi_backend.bitcask_8day.bitcask.data_root = >>>> /var/lib/riak/bitcask_8day >>>> multi_backend.bitcask_8day.bitcask.expiry = 8d >>>> multi_backend.bitcask_8day.bitcask.expiry.grace_time = 1h >>>> multi_backend.leveldb_mult.storage_backend = leveldb >>>> multi_backend.leveldb_mult.leveldb.maximum_memory.percent = 30 >>>> multi_backend.leveldb_mult.leveldb.data_root = >>>> /var/lib/riak/leveldb_mult >>>> multi_backend.bitcask_mult.storage_backend = bitcask >>>> multi_backend.bitcask_mult.bitcask.data_root = >>>> /var/lib/riak/bitcask_mult >>>> multi_backend.default = leveldb_mult >>>> >>>> $ du –h –d 1 /var/lib/riak >>>> >>>> 4.0K /var/lib/riak/riak_kv_exchange_fsm >>>> 52K /var/lib/riak/generated.configs >>>> 224K /var/lib/riak/cluster_meta >>>> 424K /var/lib/riak/ring >>>> 1.1M /var/lib/riak/kv_vnode >>>> 2.1M /var/lib/riak/bitcask_1day >>>> 3.6M /var/lib/riak/bitcask_4day >>>> 33M /var/lib/riak/yz_temp >>>> 1.1G /var/lib/riak/bitcask_2day >>>> 5.9G /var/lib/riak/yz_anti_entropy >>>> 20G /var/lib/riak/yz >>>> 24G /var/lib/riak/leveldb_mult >>>> 25G /var/lib/riak/bitcask_mult >>>> 27G /var/lib/riak/bitcask_8day >>>> 139G /var/lib/riak/anti_entropy >>>> 240G /var/lib/riak >>>> >>>> >>>> -- >>>> Rohit S. >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.basho.com_mailman_listinfo_riak-2Dusers-5Flists.basho.com&d=DwIFaQ&c=uilaK90D4TOVoH58JNXRgQ&r=qO7g3ZLnCDJ2Kp8qz13HpzC2NAX5qNCtEhoWeGshuEI&m=zcfqvyFJVK17vAft30Cv9abkK5P_f2JtRonrl0CRYnM&s=wSNoX28DcQ-CCfWp9vRUA-70z3LeWFNGHG3-LPKzAtQ&e= >>> >>> > _________________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
