Busy at work, aiming for next week.
> On Sep 1, 2016, at 8:44 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > Andrew: > HBASE-16255 has been resolved. > > Kindly provide your feedback. > > Thanks > > On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell <andrew.purt...@gmail.com> > wrote: > >> I plan to spin up a test cluster with clusterdock and try running the IT >> under a number of different scenarios. I understand snapshots have to >> function so baseline would be the calm monkey. >> >> Unless you have some other automated way for me to run the new >> functionality repeatedly, the IT is it. >> >>>> On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov <vladrodio...@gmail.com> >>> wrote: >>> >>> Not sure what do you mean, Andrew by "trying out the branch via the IT", >>> but we do not recommend running this with monkey enabled. >>> It has not been tested in a such scenario yet and frankly speaking it is >>> not supposed to work (snapshots will fail anyway and we depends on >>> snapshots) >>> >>> -Vladimir >>> >>> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell < >> andrew.purt...@gmail.com> >>> wrote: >>> >>>> Let's commit the IT to the branch, if you think the v5 patch is ready >> for >>>> commit Ted. >>>> >>>> I will be able to spend some time next week trying out the branch via >> the >>>> IT, and poking around with the new tools. After that I feel like I'll be >>>> informed enough to vote on a branch merge vote. >>>> >>>>> On Aug 19, 2016, at 12:38 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>> >>>>> IT test is provided on HBASE-16255. >>>>> >>>>> Any other comment ? >>>>> >>>>> Thanks >>>>> >>>>>> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak <dspi...@cloudera.com> >>>> wrote: >>>>>> >>>>>> Any chance for an IT test being added to the branch first? I'd love to >>>> put >>>>>> it through the paces with clusterdock to make sure it behaves well >> with >>>>>> fault injection and the like. >>>>>> >>>>>> -Dima >>>>>> >>>>>>> On Tuesday, August 2, 2016, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>> >>>>>>> Any more comments from the community on whether the merge can be >>>>>> conducted >>>>>>> ? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov < >>>>>> vladrodio...@gmail.com >>>>>>> <javascript:;>> >>>>>>> wrote: >>>>>>> >>>>>>>> Carter Shanklin posted a blog article about the feature: >>>>>>>> Some use cases and examples of a command line interface usage. >>>>>>> https://hortonworks.com/blog/coming-hdp-2-5-incremental- >>>>>> backup-restore-apache-hbase-apache-phoenix/ >>>>>>>> >>>>>>>> -Vlad >>>>>>>> >>>>>>>> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov < >>>>>>> vladrodio...@gmail.com <javascript:;> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Ok, got it. >>>>>>>>> >>>>>>>>> -Vlad >>>>>>>>> >>>>>>>>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar <e...@apache.org >>>>>>> <javascript:;>> wrote: >>>>>>>>> >>>>>>>>>> We keep the WALs which can accumulate a lot if the use case is to >>>>>> only >>>>>>>> do >>>>>>>>>> backups infrequently. This will definitely cause issues since HDFS >>>>>>> space >>>>>>>>>> will get filled up. That is why we may need an option for having >>>>>>>>>> incremental backups not used, and WAL references being deleted. >>>>>>>>>> >>>>>>>>>> Enis >>>>>>>>>> >>>>>>>>>> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov < >>>>>>>>>> vladrodio...@gmail.com <javascript:;>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Why anyone will ever need disabling incremental backups? If you >> do >>>>>>> not >>>>>>>>>> need >>>>>>>>>>> it - just run only full backups. >>>>>>>>>>> >>>>>>>>>>> -Vlad >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar <e...@apache.org >>>>>>> <javascript:;>> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks Matteo for chiming in. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi < >>>>>>>>>>> theo.berto...@gmail.com <javascript:;>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I did some review in the early beginning, but then lost track >>>>>> of >>>>>>>> the >>>>>>>>>>>>> changes. >>>>>>>>>>>>> but I'd like to give a quick review to the full code once >>>>>> people >>>>>>>>>> here >>>>>>>>>>> are >>>>>>>>>>>>> ok with getting this feature in master (2.0). >>>>>>>>>>>>> (let say we put a deadline for reviews, like 1 week for >>>>>>> reviewing >>>>>>>>>> the >>>>>>>>>>>> full >>>>>>>>>>>>> stuff after everyone agrees to get this in. just to avoid >>>>>>> holding >>>>>>>>>> this >>>>>>>>>>>> for >>>>>>>>>>>>> too long, but still enough time to have people that are >>>>>>> interested >>>>>>>>>> to >>>>>>>>>>>> look >>>>>>>>>>>>> at it. with did the same thing for MOB with a mega patch >>>>>>>>>>>>> https://reviews.apache.org/r/36391/) >>>>>>>>>>>> >>>>>>>>>>>> This sounds good. Vladimir / Ted how do you guys want to handle >>>>>>> the >>>>>>>>>>> merge? >>>>>>>>>>>> As a giant patch or a rebase of code in the branch and through >>>>>> git >>>>>>>>>> merge. >>>>>>>>>>>> >>>>>>>>>>>> We need to run a vote when the to-be-merged branched is ready. >>>>>> We >>>>>>>> can >>>>>>>>>>> set a >>>>>>>>>>>> vote timeout for at least 1 week. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> most of the code seemed isolated from the beginning, few >>>>>> changes >>>>>>>>>> here >>>>>>>>>>> and >>>>>>>>>>>>> there in the core. >>>>>>>>>>>>> so, this side of things seems ok to me. >>>>>>>>>>>>> >>>>>>>>>>>>> maybe some work to add IT tests as mentioned above, but that >>>>>>>> should >>>>>>>>>> not >>>>>>>>>>>>> take long. >>>>>>>>>>>>> >>>>>>>>>>>>> I don't know if there are already docs, but that is another >>>>>>> thing >>>>>>>> we >>>>>>>>>>> may >>>>>>>>>>>>> want to get in with the merge. >>>>>>>>>>>>> a minimal coverage at least on how to use the feature, and >>>>>> maybe >>>>>>>>>>> calling >>>>>>>>>>>> it >>>>>>>>>>>>> out as experimental? >>>>>>>>>>>>> >>>>>>>>>>>>> my main concern were around incremental backups. >>>>>>>>>>>>> I'm still not convinced around the fact that because the WALs >>>>>>>>>> contain >>>>>>>>>>>>> regions of multiple tables >>>>>>>>>>>>> the incremental backup will keep around WALs with some data >>>>>> that >>>>>>>> we >>>>>>>>>>> don't >>>>>>>>>>>>> really want in the backup (for space or maybe security >>>>>> reason). >>>>>>>>>>>>> >>>>>>>>>>>>> then there was the question about for how long should I take >>>>>>>>>>>> incrementals, >>>>>>>>>>>>> before deciding that a fresh full backup is less costly in >>>>>> terms >>>>>>>> of >>>>>>>>>>>> space. >>>>>>>>>>>>> but I think this incremental merge/compaction was a feature on >>>>>>> the >>>>>>>>>>>> roadmap >>>>>>>>>>>>> as Phase3. >>>>>>>>>>>>> which I think is ok to get later on, >>>>>>>>>>>>> maybe just call out a lifecycle example on the docs under >>>>>> "best >>>>>>>>>>>> practices". >>>>>>>>>>>> >>>>>>>>>>>> I think this will depend on the use case, and other factors like >>>>>>>>>>> bandwidth >>>>>>>>>>>> available, how much data >>>>>>>>>>>> the user is willing to lose in case of catastrophic failure and >>>>>>> how >>>>>>>>>>>> "expensive" is full backup versus >>>>>>>>>>>> incremental one. >>>>>>>>>>>> >>>>>>>>>>>> The full backup should also be useable by default, so maybe we >>>>>> can >>>>>>>>>> make >>>>>>>>>>> an >>>>>>>>>>>> option to not even keep WAL files, and completely disable >>>>>>>> incremental >>>>>>>>>>>> backups? >>>>>>>>>>>> >>>>>>>>>>>> Enis >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> has anyone interested in using backups looked at the doc in >>>>>>>>>> HBASE-7912? >>>>>>>>>>>>> is the current design of incremental backup acceptable for >>>>>>>> everyone >>>>>>>>>>>> wanting >>>>>>>>>>>>> to use this feature? >>>>>>>>>>>>> (maybe this should be a question for the @user list and not >>>>>> dev) >>>>>>>>>>>>> >>>>>>>>>>>>> is there anyone already using this feature or it is just dev >>>>>>>> testing >>>>>>>>>>> it? >>>>>>>>>>>>> to me will be interesting having a use-case/workflow example, >>>>>>>>>>>>> to see if in the real world my concerns about incremental are >>>>>>> not >>>>>>>>>>> showing >>>>>>>>>>>>> up. >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhih...@gmail.com >>>>>>> <javascript:;>> >>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Gentle ping on this subject. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The changes are mostly non-intrusive. >>>>>>>>>>>>>> >>>>>>>>>>>>>> More comments are welcome. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov < >>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Not that hard, Andrew. I will open JIRA. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Vlad >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell < >>>>>>>>>>>>>> andrew.purt...@gmail.com <javascript:;>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> How hard would it be to convert what you've been using >>>>>> to >>>>>>>> test >>>>>>>>>>> end >>>>>>>>>>>> to >>>>>>>>>>>>>> end >>>>>>>>>>>>>>>> during dev into an IT? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov < >>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If >>>>>> not, >>>>>>>> any >>>>>>>>>>> tips >>>>>>>>>>>>> on a >>>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore >>>>>>> them? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We do not have yet, but we have a lot of unit tests. >>>>>> We >>>>>>>>>>> provide 2 >>>>>>>>>>>>> API >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> backup: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Admin.getBackupAdmin >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. Command - line via hbase command. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Everything is straightforward. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Vlad >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak < >>>>>>>>>>>>> dspi...@cloudera.com <javascript:;>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If not, >>>>>>> any >>>>>>>>>> tips >>>>>>>>>>>> on >>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore >>>>>> them? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -Dima >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov < >>>>>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Sorry, wrong links: >>>>>>>>>>>>>>>>>>> These are the phases: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Phase 1: >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14030 >>>>>>>>> 14030 >>>>>>>>>>>>>>>>>>> Phase 2: >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14123 >>>>>>>>> 14123 >>>>>>>>>>>>>>>>>>> Phase 3: >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14414 >>>>>>>>> 14414 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -Vlad >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov < >>>>>>>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> These are the phases: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Phase 1: >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912 >>>>>>>>> 14030 >>>>>>>>>>>>>>>>>>>> Phase 2: >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912 >>>>>>>>> 14123 >>>>>>>>>>>>>>>>>>>> Phase 3: >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912 >>>>>>>>> 14414 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -Vlad >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar < >>>>>>>>>>>>> e...@apache.org <javascript:;>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> As you guys may already be familiar, Vladimir, >>>>>> Ted, >>>>>>>>>> Jerry >>>>>>>>>>> and >>>>>>>>>>>>>>> others >>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>> been developing the backup / restore functionality >>>>>>> in >>>>>>>> a >>>>>>>>>>>> series >>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>>>>>>>> committed in the separate branch HBASE-7912[1]. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Backup / Restore functionality is tracked as a >>>>>>> 4-phase >>>>>>>>>>>> project, >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> first two phases are complete and useable. We are >>>>>>> now >>>>>>>>>>> working >>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>> Phase 3 >>>>>>>>>>>>>>>>>>>>> items, which are mostly improvements. We think >>>>>> that >>>>>>>> the >>>>>>>>>>>> current >>>>>>>>>>>>>>> code >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> branch containing all Phase 1 and Phase 2 items, >>>>>> and >>>>>>>>>> some >>>>>>>>>>>>> Phase 3 >>>>>>>>>>>>>>>>>> items >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> useable on it's own, and we do not have to wait >>>>>> for >>>>>>>> all >>>>>>>>>> the >>>>>>>>>>>>>>>> subtickets >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> be finished to make it completely useable (as >>>>>> follow >>>>>>>> up >>>>>>>>>>>> tickets >>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>> mostly >>>>>>>>>>>>>>>>>>>>> improvements or optimizations). The improvements >>>>>> in >>>>>>>> the >>>>>>>>>>> works >>>>>>>>>>>>> are >>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>> backwards compatible with the existing stuff. >>>>>> Thus, >>>>>>> we >>>>>>>>>>> would >>>>>>>>>>>>> like >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> propose that the branch HBASE-7912 be merged into >>>>>>>>>> master. >>>>>>>>>>>> The >>>>>>>>>>>>>>> parent >>>>>>>>>>>>>>>>>>> jira >>>>>>>>>>>>>>>>>>>>> has a design doc that goes into details about the >>>>>>>>>>>>> implementation >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> design >>>>>>>>>>>>>>>>>>>>> choices in case you are interested[2]. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Most of the changes are largely non-intrusive and >>>>>>>>>> confined >>>>>>>>>>> to >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> backup subsystem. >>>>>>>>>>>>>>>>>>>>> The unit tests have been passing on manual runs >>>>>> and >>>>>>> we >>>>>>>>>>>>>>> (hortonworks) >>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>> been running the integration tests as well as some >>>>>>>> other >>>>>>>>>>>>>>> shell-based >>>>>>>>>>>>>>>>>>>>> system >>>>>>>>>>>>>>>>>>>>> tests on a forked version of the code. Most of the >>>>>>>> work >>>>>>>>>> has >>>>>>>>>>>>> been >>>>>>>>>>>>>>>>>>> reviewed >>>>>>>>>>>>>>>>>>>>> by 1, 2 or 3 committers already (mostly Ted, >>>>>> myself >>>>>>>> and >>>>>>>>>>>> Jerry). >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> What do you guys think? Is it time to call a vote? >>>>>>> Any >>>>>>>>>>>> concerns >>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>> feedback >>>>>>>>>>>>>>>>>>>>> appreciated. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> [1] >>>>>>> https://issues.apache.org/jira/browse/HBASE-7912 >>>>>>>>>>>>>>>>>>>>> [2] >>>>>>> https://issues.apache.org/jira/secure/attachment/12816339/ >>>>>> HBaseBackupAndRestore%20-0.91.pdf >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Enis >>>>>> >>>>>> >>>>>> -- >>>>>> -Dima >>
