No. I do not see a problem with your plan. But ... I would prefer to see you add servers to your cluster. Scalabilty is one of Riak's fundamental characteristics. As your database needs grow, we grow with you … just add another server and migrate some of the vnodes there.
I obviously cannot speak to your budgetary constraints. All of the engineers at Basho, I am just one, are focused upon providing you performance and features along with your scalability needs. This seems to be a situation where you might be sacrificing data integrity where another server or two would address the situation. And if 2.0 makes things better … sell the extra servers on Ebay. Matthew On Apr 8, 2014, at 6:31 PM, Edgar Veiga <edgarmve...@gmail.com> wrote: > Thanks Matthew! > > Today this situation has become unsustainable, In two of the machines I have > an anti-entropy dir of 250G... It just keeps growing and growing and I'm > almost reaching max size of the disks. > > Maybe I'll just turn off aae in the cluster, remove all the data in the > anti-entropy directory and wait for the v2 of riak. Do you see any problem > with this? > > Best regards! > > > On 8 April 2014 22:11, Matthew Von-Maszewski <matth...@basho.com> wrote: > Edgar, > > Today we disclosed a new feature for Riak's leveldb, Tiered Storage. The > details are here: > > https://github.com/basho/leveldb/wiki/mv-tiered-options > > This feature might give you another option in managing your storage volume. > > > Matthew > >> On Apr 8, 2014, at 11:07 AM, Edgar Veiga <edgarmve...@gmail.com> wrote: >> >>> It makes sense, I do a lot, and I really mean a LOT of updates per key, >>> maybe thousands a day! The cluster is experiencing a lot more updates per >>> each key, than new keys being inserted. >>> >>> The hash trees will rebuild during the next weekend (normally it takes >>> about two days to complete the operation) so I'll come back and give you >>> some feedback (hopefully good) on the next Monday! >>> >>> Again, thanks a lot, You've been very helpful. >>> Edgar >>> >>> >>> On 8 April 2014 15:47, Matthew Von-Maszewski <matth...@basho.com> wrote: >>> Edgar, >>> >>> The test I have running currently has reach 1 Billion keys. It is running >>> against a single node with N=1. It has 42G of AAE data. Here is my >>> extrapolation to compare your numbers: >>> >>> You have ~2.5 Billion keys. I assume you are running N=3 (the default). >>> AAE therefore is actually tracking ~7.5 Billion keys. You have six nodes, >>> therefore tracking ~1.25 Billion keys per node. >>> >>> Raw math would suggest that my 42G of AAE data for 1 billion keys would >>> extrapolate to 52.5G of AAE data for you. Yet you have ~120G of AAE data. >>> Is something wrong? No. My data is still loading and has experience zero >>> key/value updates/edits. >>> >>> AAE hashes get rewritten every time a user updates the value of a key. >>> AAE's leveldb is just like the user leveldb, all prior values of a key >>> accumulate in the .sst table files until compaction removes duplicates. >>> Similarly, a user delete of a key causes a delete tombstone in the AAE hash >>> tree. Those delete tombstones have to await compactions too before leveldb >>> recovers the disk space. >>> >>> AAE's hash trees rebuild weekly. I am told that the rebuild operation will >>> actually destroy the existing files and start over. That is when you >>> should see AAE space usage dropping dramatically. >>> >>> Matthew >>> >>> >>> On Apr 8, 2014, at 9:31 AM, Edgar Veiga <edgarmve...@gmail.com> wrote: >>> >>>> Thanks a lot Matthew! >>>> >>>> A little bit of more info, I've gathered a sample of the contents of >>>> anti-entropy data of one of my machines: >>>> - 44 folders with the name equal to the name of the folders in level-db >>>> dir (i.e. 393920363186844927172086927568060657641638068224/) >>>> - each folder has a 5 files (log, current, log, etc) and 5 sst_* folders. >>>> - The biggest sst folder is sst_3 with 4.3G >>>> - Inside sst_3 folder there are 1219 files name 00****.sst. >>>> - Each of the 00*****.sst files has ~3.7M >>>> >>>> Hope this info gives you some more help! >>>> >>>> Best regards, and again, thanks a lot >>>> Edgar >>>> >>>> >>>> On 8 April 2014 13:24, Matthew Von-Maszewski <matth...@basho.com> wrote: >>>> Argh. Missed where you said you had upgraded. Ok it will proceed with >>>> getting you comparison numbers. >>>> >>>> Sent from my iPhone >>>> >>>> On Apr 8, 2014, at 6:51 AM, Edgar Veiga <edgarmve...@gmail.com> wrote: >>>> >>>>> Thanks again Matthew, you've been very helpful! >>>>> >>>>> Maybe you can give me some kind of advise on this issue I'm having since >>>>> I've upgraded to 1.4.8. >>>>> >>>>> Since I've upgraded my anti-entropy data has been growing a lot and has >>>>> only stabilised in very high values... Write now my cluster has 6 >>>>> machines each one with ~120G of anti-entropy data and 600G of level-db >>>>> data. This seems to be quite a lot no? My total amount of keys is ~2.5 >>>>> Billions. >>>>> >>>>> Best regards, >>>>> Edgar >>>>> >>>>> On 6 April 2014 23:30, Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>> Edgar, >>>>> >>>>> This is indirectly related to you key deletion discussion. I made >>>>> changes recently to the aggressive delete code. The second section of >>>>> the following (updated) web page discusses the adjustments: >>>>> >>>>> https://github.com/basho/leveldb/wiki/Mv-aggressive-delete >>>>> >>>>> Matthew >>>>> >>>>> >>>>> On Apr 6, 2014, at 4:29 PM, Edgar Veiga <edgarmve...@gmail.com> wrote: >>>>> >>>>>> Matthew, thanks again for the response! >>>>>> >>>>>> That said, I'll wait again for the 2.0 (and maybe buy some bigger disks >>>>>> :) >>>>>> >>>>>> Best regards >>>>>> >>>>>> >>>>>> On 6 April 2014 15:02, Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>>> Edgar, >>>>>> >>>>>> In Riak 1.4, there is no advantage to using empty values versus deleting. >>>>>> >>>>>> leveldb is a "write once" data store. New data for a given key never >>>>>> physically overwrites old data for the same key. New data "hides" the >>>>>> old data by being in a lower level, and therefore picked first. >>>>>> >>>>>> leveldb's compaction operation will remove older key/value pairs only >>>>>> when the newer key/value is pair is part of a compaction involving both >>>>>> new and old. The new and the old key/value pairs must have migrated to >>>>>> adjacent levels through normal compaction operations before leveldb will >>>>>> see them in the same compaction. The migration could take days, weeks, >>>>>> or even months depending upon the size of your entire dataset and the >>>>>> rate of incoming write operations. >>>>>> >>>>>> leveldb's "delete" object is exactly the same as your empty JSON object. >>>>>> The delete object simply has one more flag set that allows it to also >>>>>> be removed if and only if there is no chance for an identical key to >>>>>> exist on a higher level. >>>>>> >>>>>> I apologize that I cannot give you a more useful answer. 2.0 is on the >>>>>> horizon. >>>>>> >>>>>> Matthew >>>>>> >>>>>> >>>>>> On Apr 6, 2014, at 7:04 AM, Edgar Veiga <edgarmve...@gmail.com> wrote: >>>>>> >>>>>>> Hi again! >>>>>>> >>>>>>> Sorry to reopen this discussion, but I have another question regarding >>>>>>> the former post. >>>>>>> >>>>>>> What if, instead of doing a mass deletion (We've already seen that it >>>>>>> will be non profitable, regarding disk space) I update all the values >>>>>>> with an empty JSON object "{}" ? Do you see any problem with this? I no >>>>>>> longer need those millions of values that are living in the cluster... >>>>>>> >>>>>>> When the version 2.0 of riak runs stable I'll do the update and only >>>>>>> then delete those keys! >>>>>>> >>>>>>> Best regards >>>>>>> >>>>>>> >>>>>>> On 18 February 2014 16:32, Edgar Veiga <edgarmve...@gmail.com> wrote: >>>>>>> Ok, thanks a lot Matthew. >>>>>>> >>>>>>> >>>>>>> On 18 February 2014 16:18, Matthew Von-Maszewski <matth...@basho.com> >>>>>>> wrote: >>>>>>> Riak 2.0 is coming. Hold your mass delete until then. The "bug" is >>>>>>> within Google's original leveldb architecture. Riak 2.0 sneaks around >>>>>>> to get the disk space freed. >>>>>>> >>>>>>> Matthew >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Feb 18, 2014, at 11:10 AM, Edgar Veiga <edgarmve...@gmail.com> wrote: >>>>>>> >>>>>>>> The only/main purpose is to free disk space.. >>>>>>>> >>>>>>>> I was a little bit concerned regarding this operation, but now with >>>>>>>> your feedback I'm tending to don't do nothing, I can't risk the >>>>>>>> growing of space... >>>>>>>> Regarding the overhead I think that with a tight throttling system I >>>>>>>> could control and avoid overloading the cluster. >>>>>>>> >>>>>>>> Mixed feelings :S >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 18 February 2014 15:45, Matthew Von-Maszewski <matth...@basho.com> >>>>>>>> wrote: >>>>>>>> Edgar, >>>>>>>> >>>>>>>> The first "concern" I have is that leveldb's delete does not free disk >>>>>>>> space. Others have executed mass delete operations only to discover >>>>>>>> they are now using more disk space instead of less. Here is a >>>>>>>> discussion of the problem: >>>>>>>> >>>>>>>> https://github.com/basho/leveldb/wiki/mv-aggressive-delete >>>>>>>> >>>>>>>> The link also describes Riak's database operation overhead. This is a >>>>>>>> second "concern". You will need to carefully throttle your delete >>>>>>>> rate or the overhead will likely impact your production throughput. >>>>>>>> >>>>>>>> We have new code to help quicken the actual purge of deleted data in >>>>>>>> Riak 2.0. But that release is not quite ready for production usage. >>>>>>>> >>>>>>>> >>>>>>>> What do you hope to achieve by the mass delete? >>>>>>>> >>>>>>>> Matthew >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Feb 18, 2014, at 10:29 AM, Edgar Veiga <edgarmve...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Sorry, forgot that info! >>>>>>>>> >>>>>>>>> It's leveldb. >>>>>>>>> >>>>>>>>> Best regards >>>>>>>>> >>>>>>>>> >>>>>>>>> On 18 February 2014 15:27, Matthew Von-Maszewski <matth...@basho.com> >>>>>>>>> wrote: >>>>>>>>> Which Riak backend are you using: bitcask, leveldb, multi? >>>>>>>>> >>>>>>>>> Matthew >>>>>>>>> >>>>>>>>> >>>>>>>>> On Feb 18, 2014, at 10:17 AM, Edgar Veiga <edgarmve...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> > Hi all! >>>>>>>>> > >>>>>>>>> > I have a fairly trivial question regarding mass deletion on a riak >>>>>>>>> > cluster, but firstly let me give you just some context. My cluster >>>>>>>>> > is running with riak 1.4.6 on 6 machines with a ring of 256 nodes >>>>>>>>> > and 1Tb ssd disks. >>>>>>>>> > >>>>>>>>> > I need to execute a massive object deletion on a bucket, I'm >>>>>>>>> > talking of ~1 billion keys (The object average size is ~1Kb). I >>>>>>>>> > will not retrive the keys from riak because a I have a file with >>>>>>>>> > all of them. I'll just start a script that reads them from the file >>>>>>>>> > and triggers an HTTP DELETE for each one. >>>>>>>>> > The cluster will continue running on production with a quite high >>>>>>>>> > load serving all other applications, while running this deletion. >>>>>>>>> > >>>>>>>>> > My question is simple, do I need to have any kind of extra concerns >>>>>>>>> > regarding this action? Do you advise me on taking special attention >>>>>>>>> > to any kind of metrics regarding riak or event the servers where >>>>>>>>> > it's running? >>>>>>>>> > >>>>>>>>> > Best regards! >>>>>>>>> > _______________________________________________ >>>>>>>>> > riak-users mailing list >>>>>>>>> > riak-users@lists.basho.com >>>>>>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >> > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
