[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16909365#comment-16909365
]
Sean Busbey commented on HBASE-22749:
-------------------------------------
h2. region sizing - splitting, normalizers, etc
Need to expressly state wether or not this change to per-region accounting
plans to alter the current assumptions that use of the feature means that the
MOB data isn’t counted when determining region size for decisions to normalize
or split.
h2. write amplification
Current description of the unified compactor’s handling of MOB data doesn’t
include anything about doing the kind of mob file partitioning that was
previously done. I think this will behave a lot like the example from Section
3.1.1 in the MOB Design v5 document you reference, specifically where MOB stuff
is segregated in a dedicated CF. We still end up getting unbounded write
amplification.
Consider this use case, which I think is in line with the assumptions laid out
in both your description and in the MOB Design v5 document:
* Table with 50k regions
* MOB values that are 300KiB
* No updates, no deletes
* periodic flushes set to 6 hours
* periodic major and mob compaction set to weekly
* infrequent writes (slow enough so that only periodic flushes happen, but
enough that all regions have a mob write)
Under the current MOB implementation with a monthly partition policy, I can
reason that:
* we’ll be generating 200k hfiles in the mob directory per day due to periodic
flushes
* at the first week we’ll have 1.4m new hfiles, which we’ll compact probably
into a low-double-digit number of hfiles
* at the second week, we’ll have 1.4m new hfiles plus the results of the first
compaction. we will probably compact this into a low-double-digit number of
hfiles
* at the third week, same thing again
* on the fourth week, same thing again
* After that fourth week things repeat, but the files generated will be in a
new partition and so anything from prior won’t be rewritten again.
In the steady state:
* we should have a number of hfiles that stays under the limits of HDFS
* for a given MOB value, we should only write it to HDFS no more than 5 times
(flush + between 1 and 4 compactions)
So that means we have a write amplification of ~5x regardless of splits or
merges from normalization.
For the new design I don’t think there’s currently any bound. If I use the
default compaction strategy:
* We’ll still be generating 200k hfiles per day
* at the first week we’ll have 1.4m new hfiles which we’ll compact to 50k
hfiles.
* at the end of second week we’ll have 1.4m new hfiles + the existing 50k
hfiles, and we’ll compact to 50k hfiles
* third week, same thing
* forth week, same thing
* this will repeat until each of the 50k files hits 10 GiB - 20 GiB depending
on configs (~35-70k cells)
At the extreme of exactly 1 mob value per region per periodic flush that would
mean 1-2 thousand weeks. Splits over that time period would mean probably we’d
keep rewriting indefinitely. So the amount of amplification is essentially
going to be driven by the periodicity of the mob compaction chore.
With default configs we can still get memstores that have ~1GiB of MOB values
and still only do periodic flushes, so this can remain a non-trivial amount of
load on HDFS.
If we enable partial major mob compaction we’d avoid writing the values
repeatedly, but we’d against HDFS limitations in ~10 days.
h2. MOB compaction request chore and Partial major mob compaction
It’s a bit confusing going through the “Partial major MOB compaction” section
where it currently is in the write up. As I understand things, you’re
essentially describing a strategy for the process that has to pick particular
regions to issue major compaction requests against instead of just requesting
the whole table be compacted. Since this is an optimization of cluster IO use
that’s possible _once we have per-region accounting and maintenance of MOB
data_ I think it’d be clearer if it was in a section _after_ you describe the
“scalable MOB compactions” stuff.
instead of starting that section off with the description of the compaction
request chore, you can explain the accounting changes to store maintenance, the
resulting changes to cleaning, and then end with the explanation about how
folks won’t have to schedule maintenance tasks themselves with a section that’s
labeled as the description of the “MOB Compaction Request Chore” and include
there the description fo the region prioritization strategy. Another good
strategy to mention there is prioritizing the store files we know haven’t been
converted to include accounting information the cleaner needs.
h2. split tracking for the above
Could we do this with entries in hbase:meta or a journal instead of individual
files? It’s going to get very messy when there are tables with
tens-of-thousands or hundreds-of-thousands of regions.
h2. metrics needed
Some metrics that would be useful while reading the design:
* compaction request chore, esp when using partial major mob comapction. time
to evaluate, number of regions with/without siblings, number of regions selected
* mob cleaner chore - time to evaluate, # mob files referenced, # mob files not
referenced, # old store files, skipped store files (e.g. from bulkload)
* unified compact/flush - number of new reference files / number of passed
through reference files / number of no longer needed references (not cells,
specifically to have an idea of the size of reference metadata we’re keeping
around)
h2. performance considerations section
include a description of tests to run to compare before and after performance.
e.g. using Load Test Tool to run X records for a single node, 5 node, 10 node
with values of ~100KiB, ~1MiB, ~10MiB
{code}
hbase ltt -mob_threshold 102400 -generator \
org.apache.hadoop.hbase.util.LoadTestDataGeneratorWithMOB:example_mob:102400:104857
\
-num_keys 10000 -write 3:1024 -tn table_1 -families plain,example_mob
{code}
h2. MOB 2.0 Compaction for small MOB (below 50KiB) - remove since we said it is
a non-goal
h2. MOB 2.0 functional testing (stress tool / fault injections)
* should be framed in terms of using ltt to do the data load/reading. something
like the above sample command.
* What about Chaos Monkey for HDFS failures / RS failures / Master failures? if
it’s not necessary, include reasoning on why?
h2. tools updates
* Don’t snapshots need to change? Intuitively I would expect us to need to walk
the hfiles from the snapshot to get the list of mob files to include, rather
than just including everything in the mobdir?
* Similarly, doesn’t incremental backup/restore need to adapt?
* Prior to this refactoring I would have the offline compaction tool to offload
my non-mob data compaction and I’d have the external mob compaction tool to
offload my mob data. What about updating the offline compaction tool to
optionally handle mob data needed?
* HFile Pretty Printer tool needs some updates. should include something about
the mob reference metadata (like a count in normal mode and a listing in
verbose maybe?). the mob integrity check option needs to include validating
that all the references found in cells are also found in the file metadata
> Distributed MOB compactions
> ----------------------------
>
> Key: HBASE-22749
> URL: https://issues.apache.org/jira/browse/HBASE-22749
> Project: HBase
> Issue Type: New Feature
> Components: mob
> Reporter: Vladimir Rodionov
> Assignee: Vladimir Rodionov
> Priority: Major
> Attachments: HBase-MOB-2.0-v1.pdf, HBase-MOB-2.0-v2.1.pdf,
> HBase-MOB-2.0-v2.pdf
>
>
> There are several drawbacks in the original MOB 1.0 (Moderate Object
> Storage) implementation, which can limit the adoption of the MOB feature:
> # MOB compactions are executed in a Master as a chore, which limits
> scalability because all I/O goes through a single HBase Master server.
> # Yarn/Mapreduce framework is required to run MOB compactions in a scalable
> way, but this won’t work in a stand-alone HBase cluster.
> # Two separate compactors for MOB and for regular store files and their
> interactions can result in a data loss (see HBASE-22075)
> The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible
> implementation, which is free of the above drawbacks and can be used as a
> drop in replacement in existing MOB deployments. So, these are design goals
> of a MOB 2.0:
> # Make MOB compactions scalable without relying on Yarn/Mapreduce framework
> # Provide unified compactor for both MOB and regular store files
> # Make it more robust especially w.r.t. to data losses.
> # Simplify and reduce the overall MOB code.
> # Provide 100% compatible implementation with MOB 1.0.
> # No migration of data should be required between MOB 1.0 and MOB 2.0 - just
> software upgrade.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
