[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614916#comment-16614916
]
Josh Elser commented on HBASE-20952:
------------------------------------
{quote}Yes this is what I really want to discuss, not something like whether we
should use WALInfo or WALIdentity.
{quote}
We're in agreement... API just gives something tangible to look at when
thinking about requirements of the system. I think it's helpful, that's fine if
you don't :). Having a pissing match about the name of a class at this point is
a waste of time.
{quote}it seems to me that we will only have one stream opened forever for a
RS, then how do we drop the old edits after flush
{quote}
No, the plan would be to mimic the RS "rolling" logs in Ratis as well. This was
discussed over on the Ratis side. Seems to not have made it here – sorry.
{quote}In the old time we will roll the wal writer, and it is done by RS, so
closing the wal file is not enough, as the RS will try to open a new one and
write to it. That's why we need to rename the wal directory
{quote}
Crap. The current vision has a half-fix for this: just like we can do for the
Log used by one RS, we can list all Logs used by this RS and force them closed.
We don't (presently) have a mechanism spec'ed out in the LogService to prevent
a client from creating new Logs. We will have a "centralized" service that
manages the metadata for the Ratis logs in the system, so I think we _could_ do
something equivalent in the LogService (prevent a specific client from creating
a new Log). Need to think about that some more.
{quote}And for the FileSystem, we will use multi wal to increase the
performance, and the logic is messed up with WALProvider.
{quote}
Tell me more, please.
{quote}Does ratis still need multi wal to increase the performance? And if not,
what's the plan? We need to refactor the multi wal related code, to not work
against the WALProvider but something with the FileSystem related stuffs
directly?
{quote}
It's a goal that MultiWAL can work with any other WAL implementation, but not a
"priority 0" kind of concern to me.
{quote} For FileSystem based wals, it is just a directory on a remote cluster,
for example, "hdfs://cluster-name/path". We need to find a way to describe
other log systems.
{quote}
Got it. Yeah, this is another hole right now. [~rajeshbabu] and
[~sergey.soldatov] would probably be interested in this one; they've been
thinking about what logic we'll need to build in LogService (generic) and what
logic we'll have to build in HBase (specific).
Probably the biggest open question around this is how HBase will find the
LogService "metadata" service (e.g. the aforementioned piece that gives DDL
operations around Logs). Should the LogService provide some mechanism to
publish this? Or just leave it as an example to the implementation (e.g. HBase
master handles it)?
Thinking through the basics: an HDFS URI is logically equivalent to what we'd
be using in the LogService: HDFS UI gives us a nameservice and the unique name
of a file. For the LogService, we'd have the Raft quorum for the discovery
service and the name of the log to create. Needs some more thought around a
more elegant abstraction, I think...
> Re-visit the WAL API
> --------------------
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
> Reporter: Josh Elser
> Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup&restore. Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B&R doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
