Can someone review this pull request?
https://github.com/apache/hbase/pull/3359
This change changes meta information for backup, if not part of hbase
3.0.0. It might have a lot of additional work to be put into executing the
above mentioned plan.
---
Mallikarjun
On Thu, Feb 11, 2021 at 5:36 PM Mallikarjun
wrote:
> Slight modification to previous version --> https://ibb.co/Nttx3J1
>
> ---
> Mallikarjun
>
>
> On Thu, Feb 11, 2021 at 8:12 AM Mallikarjun
> wrote:
>
>> Inline Reply
>>
>> On Wed, Feb 3, 2021 at 6:44 AM Sean Busbey wrote:
>>
>>> Hi Mallikarjun,
>>>
>>> Those goals sound worthwhile.
>>>
>>> Do you have a flow chart similar to the one you posted for the current
>>> system but for the proposed solution?
>>>
>>
>> This is what I am thinking --> https://ibb.co/KmH6Cwv
>>
>>
>>>
>>> How much will we need to change our existing test coverage to accommodate
>>> the proposed solution?
>>>
>>
>> Of the 38 tests, it looks like we might have to change a couple only.
>> Will have to add more tests to cover parallel backup scenarios.
>>
>>
>>>
>>> How much will we need to update the existing reference guide section?
>>>
>>>
>> Probably nothing. Interface as such will not change.
>>
>>
>>>
>>> On Sun, Jan 31, 2021, 04:59 Mallikarjun
>>> wrote:
>>>
>>> > Bringing up this thread.
>>> >
>>> > On Mon, Jan 25, 2021, 3:38 PM Viraj Jasani wrote:
>>> >
>>> > > Thanks, the image is visible now.
>>> > >
>>> > > > Since I wanted to open this for discussion, did not consider
>>> placing it
>>> > > in
>>> > > *hbase/dev_support/design-docs*.
>>> > >
>>> > > Definitely, only after we come to concrete conclusion with the
>>> reviewer,
>>> > we
>>> > > should open up a PR. Until then this thread is anyways up for
>>> discussion.
>>> > >
>>> > >
>>> > > On Mon, 25 Jan 2021 at 1:58 PM, Mallikarjun <
>>> mallik.v.ar...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Hope this link works --> https://ibb.co/hYjRpgP
>>> > > >
>>> > > > Inline reply
>>> > > > On Mon, Jan 25, 2021 at 1:16 PM Viraj Jasani
>>> > wrote:
>>> > > >
>>> > > > > Hi,
>>> > > > >
>>> > > > > Still not available :)
>>> > > > > The attachments don’t work on mailing lists. You can try
>>> uploading
>>> > the
>>> > > > > attachment on some public hosting site and provide the url to the
>>> > same
>>> > > > > here.
>>> > > > >
>>> > > > > Since I am not aware of the contents, I cannot confirm right
>>> away but
>>> > > if
>>> > > > > the reviewer feels we should have the attachment on our github
>>> repo:
>>> > > > > hbase/dev-support/design-docs , good to upload the content there
>>> > later.
>>> > > > For
>>> > > > > instance, pdf file can contain existing design and new design
>>> > diagrams
>>> > > > and
>>> > > > > talk about pros and cons etc once we have things finalized.
>>> > > > >
>>> > > > >
>>> > > > Since I wanted to open this for discussion, did not consider
>>> placing it
>>> > > in
>>> > > > *hbase/dev_support/design-docs*.
>>> > > >
>>> > > >
>>> > > > >
>>> > > > > On Mon, 25 Jan 2021 at 12:13 PM, Mallikarjun <
>>> > mallik.v.ar...@gmail.com
>>> > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Attached as image. Please let me know if it is availabe now.
>>> > > > > >
>>> > > > > > ---
>>> > > > > > Mallikarjun
>>> > > > > >
>>> > > > > >
>>> > > > > > On Mon, Jan 25, 2021 at 10:32 AM Sean Busbey <
>>> bus...@apache.org>
>>> > > > wrote:
>>> > > > > >
>>> > > > > >> Hi!
>>> > > > > >>
>>> > > > > >> Thanks for the write up. unfortunately, your image for the
>>> > existing
>>> > > > > >> design didn't come through. Could you post it to some host and
>>> > link
>>> > > it
>>> > > > > >> here?
>>> > > > > >>
>>> > > > > >> On Sun, Jan 24, 2021 at 3:12 AM Mallikarjun <
>>> > > mallik.v.ar...@gmail.com
>>> > > > >
>>> > > > > >> wrote:
>>> > > > > >> >
>>> > > > > >> > Existing Design:
>>> > > > > >> >
>>> > > > > >> >
>>> > > > > >> >
>>> > > > > >> > Problem 1:
>>> > > > > >> >
>>> > > > > >> > With this design, Incremental and Full backup can't be run
>>> in
>>> > > > parallel
>>> > > > > >> and leading to degraded RPO's in case Full backup is of longer
>>> > > > duration
>>> > > > > esp
>>> > > > > >> for large tables.
>>> > > > > >> >
>>> > > > > >> > Example:
>>> > > > > >> > Expectation: Say you have a big table with 10 TB and your
>>> RPO is
>>> > > 60
>>> > > > > >> minutes and you are allowed to ship the remote backup with 800
>>> > Mbps.
>>> > > > And
>>> > > > > >> you are allowed to take Full Backups once in a week and rest
>>> of
>>> > them
>>> > > > > should
>>> > > > > >> be incremental backups
>>> > > > > >> >
>>> > > > > >> > Shortcoming: With the above design, one can't run parallel
>>> > backups
>>> > > > and
>>> > > > > >> whenever there is a full backup running (which takes roughly
>>> 25
>>> > > hours)
>>> > > > > you
>>> > > > > >> are not allowed to take incremental backups and that would be
>>> a
>>> > > breach
>>> > > > > in
>>> > > > > >> your RPO.
>>> > > > > >> >
>>> > > > > >> > Proposed Solution: Barring some
