[
https://issues.apache.org/jira/browse/HBASE-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768703#comment-15768703
]
Enis Soztutar commented on HBASE-17018:
---------------------------------------
Thanks for entertaining my suggestion.
bq. On our case we launch ~1K containers per second. If we write 100 metrics
each, the total volume written into HBase is considerable
HBase in the end will end up writing these events to it's own WAL's on HDFS. So
in terms of scalability, you should be able to achieve HBase throughput much
much easier, since HBase is doing a lot more work (RPC, sorting data, flushing
to disk, compaction, etc).
bq. For large deployments that means that there could be hundreds of parallel
writers.
That should be fine for HDFS, as long as you have 1 writer per application,
rather than 1 writer per task.
bq. It would essentially double the hdfs requirement for the storage
I was thinking that you would delete the records once the reader has persisted
them to HBase. If the application writer is dead, some other application writer
eventually finishes the persisting to HBase (because WALs are already there in
HDFS). For example, HBase keeps rolling the WAL to a new file every ~100MB's.
Then the whole file is deleted once we determine that it is not needed anymore.
bq. Would the reader still query hbase only and return no data if hbase is
missing the data?
I think that is determined by the requirements for ATS. You have to determine
the "commit point" and the read point semantics. For example, you can have it
so that commit point is the HDFS write. Once it is complete, you ACK the write
which means HBase write will be "eventual consitent" with the benefit of not
depending on HBase availability. Or you can make it so that, the commit point
is HDFS write + wait for Hbase write for 30seconds. In this case, you wait for
HBase for 30 sec, but still ACK the write once it hits HDFS after timeout. It
also depends on whether you need read-after-write semantics or not. If so,
maybe you do a in-memory cache for stuff waiting to be written to HBase. Not
sure on ATS requirements.
> Spooling BufferedMutator
> ------------------------
>
> Key: HBASE-17018
> URL: https://issues.apache.org/jira/browse/HBASE-17018
> Project: HBase
> Issue Type: New Feature
> Reporter: Joep Rottinghuis
> Attachments: HBASE-17018.master.001.patch,
> HBASE-17018.master.002.patch, HBASE-17018.master.003.patch,
> HBASE-17018.master.004.patch,
> HBASE-17018SpoolingBufferedMutatorDesign-v1.pdf, YARN-4061 HBase requirements
> for fault tolerant writer.pdf
>
>
> For Yarn Timeline Service v2 we use HBase as a backing store.
> A big concern we would like to address is what to do if HBase is
> (temporarily) down, for example in case of an HBase upgrade.
> Most of the high volume writes will be mostly on a best-effort basis, but
> occasionally we do a flush. Mainly during application lifecycle events,
> clients will call a flush on the timeline service API. In order to handle the
> volume of writes we use a BufferedMutator. When flush gets called on our API,
> we in turn call flush on the BufferedMutator.
> We would like our interface to HBase be able to spool the mutations to a
> filesystems in case of HBase errors. If we use the Hadoop filesystem
> interface, this can then be HDFS, gcs, s3, or any other distributed storage.
> The mutations can then later be re-played, for example through a MapReduce
> job.
> https://reviews.apache.org/r/54882/
> For design of SpoolingBufferedMutatorImpl see
> https://docs.google.com/document/d/1GTSk1Hd887gGJduUr8ZJ2m-VKrIXDUv9K3dr4u2YGls/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
