Thanks Josh that was helpful. I'll start doing some of my own research around these topics and look into that Ratis ticket. Much appreciated!
On Mon, Aug 6, 2018 at 8:04 AM, Josh Elser <[email protected]> wrote: > Yup, replication is a big one to "unravel". Repeating myself from a branch > in the thread, but I'd expect some initial suggestions on what a new API > could be this week. Certainly the first draft won't be the final -- would > be great to get your input after your AsyncWAL work, Duo. > > Using AWS SimpleQueryService, or much anything else, would be great. I > want to make sure that, while we try to "scratch this one itch", we pave > the way for whatever else folks want to experiment with. > > > On 8/4/18 5:10 AM, 张铎(Duo Zhang) wrote: > >> Yes, maybe we could write WAL to SQS and HFile to S3, then we can deploy >> HBase on AWS without any local storage volumes... >> >> But we also need a good abstraction for Replication, as the current design >> is file based... >> >> 2018-07-27 1:28 GMT+08:00 Zach York <[email protected]>: >> >> I would REALLY hope that the WAL interface/API changes would go into >>> master >>> even if the feature work for Ratis is going in a feature branch. Not only >>> would this enable other backends to be developed in parallel with the >>> Ratis >>> solution if there are other good fits for a non-HDFS WAL, but also it >>> would >>> save the burden of having to rebase these core changes onto the latest >>> master to maintain compatibility. I'm assuming the Ratis portion of the >>> code would be mostly new files so these would be less of a concern. >>> >>> On Thu, Jul 26, 2018 at 9:24 AM, Josh Elser <[email protected]> wrote: >>> >>> On 7/26/18 1:00 AM, Stack wrote: >>>> >>>> All this said, I'd like to start moving toward the point where we start >>>>> >>>>>> breaking out this work into a feature-branch off of master and start >>>>>> building code. My hope is that this is amenable to everyone, with the >>>>>> acknowledge that the Ratis work is considered "experimental" and not >>>>>> an >>>>>> attempt to make all of HBase use Ratis-backed WALs. >>>>>> >>>>>> >>>>>> Go for it. >>>>>> >>>>> >>>>> The branch would have WAL API changes only or would it include Ratis >>>>> WAL >>>>> dev? (If the latter, would that be better done over on Ratis project?). >>>>> >>>>> >>>> I think we would start with WAL API changes, get those "blessed", and >>>> >>> then >>> >>>> continue Ratis WAL dev after that. >>>> >>>> >>> >>
