Thanks Tim! I've seen both problems myself (independent of Tim I should add, even though he helps us here) and continue to see them at customers almost every month.
1) Hard to determine proper pre-split points: Everyone I meet either just guesses or writes their own little program that basically does the same as the little program another customer wrote. So I definitely think having such a tool would be useful. We could probably make it pretty easy. For people that have already written a BulkLoad using the HFileOutputFormat2 we can just ignore the values and focus on keys only. Ideally (Tim, correct me if I'm wrong) there should be no code changes for the user. 2) Approaches to fix badly skewed tables: Definitely. My use-case here is different than Tim's. Tim can afford to write into a new table which my customers usually can't easily do. So for those kinds of users it would be great to have two things: a) An easier way (the current UI shows lots of stats etc. but doesn't help much with "insight" - so you need the knowledge of HBase's inner workings to understand what those numbers mean b) A defined way to get a HBase table into a good state. As you said: A more aggressive Normalizer sounds like a great solution to me. As the Normalizer is set on the cluster level I'm not sure what the best solution would be here. It'd be great if we could start a one-off Normalizer from the shell using a different class. I also don't see an obvious way to pass in options to an invocation of the Normalizer. The Javadoc for RegionNormalizer says: "Please note that overly aggressive normalization rules (attempting to make all regions perfectly equal in size) could potentially lead to "split/merge storms" and I agree with that. This "new" Normalizer shouldn't be the default. Cheers, Lars On Sun, Apr 22, 2018 at 1:35 AM, Tim <timrobertson...@gmail.com> wrote: > Definitely not normalizer - wasn’t enabled at that point. I believe it > happened in 2 places because of: > > 1) poorly implemented bulk load pre-split strategy > 2) integer keys and lots of row deletions > > Seem plausible? > > Thanks > Tim, > Sent from my iPhone > > > On 21 Apr 2018, at 19:29, Ted Yu <yuzhih...@gmail.com> wrote: > > > > From your first email: > > > > bq. including some with many regions in the KB size > > > > Do you know if the above was result of the operation(s) from normalizer ? > > Since assuming you use standard max hfile size, there shouldn't be such > > small regions. > > > > Cheers > > > > On Sat, Apr 21, 2018 at 10:18 AM, Tim Robertson < > timrobertson...@gmail.com> > > wrote: > > > >> Thanks Ted, > >> > >> I should have been explicit - for the cases I've been working with they > can > >> make their apps effectively go "read-only" for this house keeping > step. At > >> the end a change of app config or a couple of table name changes (short > >> outage) would be needed. > >> > >> I've been using the SimpleNormalizer in 1.2.0 (CDH 5.12+) - I'll dig > into > >> the recent changes. I had to run several iterations of small region > >> merging, plus a few iterations of SimpleNormalization to get a decent > >> result which took a long time (days). On Normalizer - I had wondered if > an > >> approach of determining a good set of splits up front might be portable > >> into a Normalizer implementation. > >> > >> I suspect a one time rewrite is cheaper than normalization when a table > is > >> in really bad shape. > >> > >> Thanks again, > >> Tim > >> > >> > >> > >> > >>> On Sat, Apr 21, 2018 at 6:59 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >>> > >>> Looking at proposed flow, have you considered the new data coming in > >>> between steps #a and #d ? > >>> > >>> Also, how would client application switch between the original table > and > >>> the new table ? > >>> > >>> BTW since you mentioned SimpleNormalizer, which release are you using > >> (just > >>> want to see if all recent fixes to SimpleNormalizer were in the version > >> you > >>> use) ? > >>> > >>> Cheers > >>> > >>> On Sat, Apr 21, 2018 at 9:48 AM, Tim Robertson < > >> timrobertson...@gmail.com> > >>> wrote: > >>> > >>>> Hi folks > >>>> > >>>> Recently I've seen a few clusters with badly unbalanced tables, > >> including > >>>> some with many regions in the KB size. It seems it is easy to overlook > >>> this > >>>> in ops. > >>>> > >>>> Understandably SimpleNormalizer does a fairly poor job at addressing > >>> this - > >>>> takes a long time, doesn't aggressively merge small regions, eagerly > >>> splits > >>>> well sized regions if many small ones exist etc. It works well if > >> enabled > >>>> on a well set up table though. > >>>> > >>>> I have been exploring approaches to tackle: > >>>> 1) determining region splits for a one time bulk load into a presplit > >>>> table[1] and > >>>> 2) approaches to fixing really badly skewed tables. > >>>> > >>>> I was thinking of creating a Jira which I'd assign to myself to add a > >>>> utility tool that would: > >>>> > >>>> a) read the HFiles for a table (optionally performing a MC first to > >>>> discard old edits) > >>>> b) analyze the block headers and determine splits that would take you > >>>> back to regions at e.g. 80% hbase.hregion.max.filesize > >>>> c) create a new pre-split table > >>>> d) run a table copy (or bulkload?) > >>>> > >>>> Does such a thing exist anywhere and I'm just missing it, or does > >> anyone > >>>> know of a better approach please? > >>>> > >>>> Thoughts, criticism, requests very welcome. > >>>> > >>>> Thanks, > >>>> Tim > >>>> > >>>> [1] > >>>> https://github.com/opencore/hbase-bulk-load-balanced/blob/ > >>>> master/src/test/java/com/opencore/hbase/example/ExampleUsageTest.java > >>>> > >>> > >> >
