Andrew: Do you think you would have some time this week ? Thanks
On Thu, Sep 1, 2016 at 8:47 AM, Andrew Purtell <andrew.purt...@gmail.com> wrote: > Busy at work, aiming for next week. > > > On Sep 1, 2016, at 8:44 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > Andrew: > > HBASE-16255 has been resolved. > > > > Kindly provide your feedback. > > > > Thanks > > > > On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell < > andrew.purt...@gmail.com> > > wrote: > > > >> I plan to spin up a test cluster with clusterdock and try running the IT > >> under a number of different scenarios. I understand snapshots have to > >> function so baseline would be the calm monkey. > >> > >> Unless you have some other automated way for me to run the new > >> functionality repeatedly, the IT is it. > >> > >>>> On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov < > vladrodio...@gmail.com> > >>> wrote: > >>> > >>> Not sure what do you mean, Andrew by "trying out the branch via the > IT", > >>> but we do not recommend running this with monkey enabled. > >>> It has not been tested in a such scenario yet and frankly speaking it > is > >>> not supposed to work (snapshots will fail anyway and we depends on > >>> snapshots) > >>> > >>> -Vladimir > >>> > >>> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell < > >> andrew.purt...@gmail.com> > >>> wrote: > >>> > >>>> Let's commit the IT to the branch, if you think the v5 patch is ready > >> for > >>>> commit Ted. > >>>> > >>>> I will be able to spend some time next week trying out the branch via > >> the > >>>> IT, and poking around with the new tools. After that I feel like I'll > be > >>>> informed enough to vote on a branch merge vote. > >>>> > >>>>> On Aug 19, 2016, at 12:38 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >>>>> > >>>>> IT test is provided on HBASE-16255. > >>>>> > >>>>> Any other comment ? > >>>>> > >>>>> Thanks > >>>>> > >>>>>> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak <dspi...@cloudera.com> > >>>> wrote: > >>>>>> > >>>>>> Any chance for an IT test being added to the branch first? I'd love > to > >>>> put > >>>>>> it through the paces with clusterdock to make sure it behaves well > >> with > >>>>>> fault injection and the like. > >>>>>> > >>>>>> -Dima > >>>>>> > >>>>>>> On Tuesday, August 2, 2016, Ted Yu <yuzhih...@gmail.com> wrote: > >>>>>>> > >>>>>>> Any more comments from the community on whether the merge can be > >>>>>> conducted > >>>>>>> ? > >>>>>>> > >>>>>>> Thanks > >>>>>>> > >>>>>>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov < > >>>>>> vladrodio...@gmail.com > >>>>>>> <javascript:;>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Carter Shanklin posted a blog article about the feature: > >>>>>>>> Some use cases and examples of a command line interface usage. > >>>>>>> https://hortonworks.com/blog/coming-hdp-2-5-incremental- > >>>>>> backup-restore-apache-hbase-apache-phoenix/ > >>>>>>>> > >>>>>>>> -Vlad > >>>>>>>> > >>>>>>>> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov < > >>>>>>> vladrodio...@gmail.com <javascript:;> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Ok, got it. > >>>>>>>>> > >>>>>>>>> -Vlad > >>>>>>>>> > >>>>>>>>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar <e...@apache.org > >>>>>>> <javascript:;>> wrote: > >>>>>>>>> > >>>>>>>>>> We keep the WALs which can accumulate a lot if the use case is > to > >>>>>> only > >>>>>>>> do > >>>>>>>>>> backups infrequently. This will definitely cause issues since > HDFS > >>>>>>> space > >>>>>>>>>> will get filled up. That is why we may need an option for having > >>>>>>>>>> incremental backups not used, and WAL references being deleted. > >>>>>>>>>> > >>>>>>>>>> Enis > >>>>>>>>>> > >>>>>>>>>> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov < > >>>>>>>>>> vladrodio...@gmail.com <javascript:;>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Why anyone will ever need disabling incremental backups? If you > >> do > >>>>>>> not > >>>>>>>>>> need > >>>>>>>>>>> it - just run only full backups. > >>>>>>>>>>> > >>>>>>>>>>> -Vlad > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar < > e...@apache.org > >>>>>>> <javascript:;>> > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Thanks Matteo for chiming in. > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi < > >>>>>>>>>>> theo.berto...@gmail.com <javascript:;>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I did some review in the early beginning, but then lost track > >>>>>> of > >>>>>>>> the > >>>>>>>>>>>>> changes. > >>>>>>>>>>>>> but I'd like to give a quick review to the full code once > >>>>>> people > >>>>>>>>>> here > >>>>>>>>>>> are > >>>>>>>>>>>>> ok with getting this feature in master (2.0). > >>>>>>>>>>>>> (let say we put a deadline for reviews, like 1 week for > >>>>>>> reviewing > >>>>>>>>>> the > >>>>>>>>>>>> full > >>>>>>>>>>>>> stuff after everyone agrees to get this in. just to avoid > >>>>>>> holding > >>>>>>>>>> this > >>>>>>>>>>>> for > >>>>>>>>>>>>> too long, but still enough time to have people that are > >>>>>>> interested > >>>>>>>>>> to > >>>>>>>>>>>> look > >>>>>>>>>>>>> at it. with did the same thing for MOB with a mega patch > >>>>>>>>>>>>> https://reviews.apache.org/r/36391/) > >>>>>>>>>>>> > >>>>>>>>>>>> This sounds good. Vladimir / Ted how do you guys want to > handle > >>>>>>> the > >>>>>>>>>>> merge? > >>>>>>>>>>>> As a giant patch or a rebase of code in the branch and through > >>>>>> git > >>>>>>>>>> merge. > >>>>>>>>>>>> > >>>>>>>>>>>> We need to run a vote when the to-be-merged branched is ready. > >>>>>> We > >>>>>>>> can > >>>>>>>>>>> set a > >>>>>>>>>>>> vote timeout for at least 1 week. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> most of the code seemed isolated from the beginning, few > >>>>>> changes > >>>>>>>>>> here > >>>>>>>>>>> and > >>>>>>>>>>>>> there in the core. > >>>>>>>>>>>>> so, this side of things seems ok to me. > >>>>>>>>>>>>> > >>>>>>>>>>>>> maybe some work to add IT tests as mentioned above, but that > >>>>>>>> should > >>>>>>>>>> not > >>>>>>>>>>>>> take long. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I don't know if there are already docs, but that is another > >>>>>>> thing > >>>>>>>> we > >>>>>>>>>>> may > >>>>>>>>>>>>> want to get in with the merge. > >>>>>>>>>>>>> a minimal coverage at least on how to use the feature, and > >>>>>> maybe > >>>>>>>>>>> calling > >>>>>>>>>>>> it > >>>>>>>>>>>>> out as experimental? > >>>>>>>>>>>>> > >>>>>>>>>>>>> my main concern were around incremental backups. > >>>>>>>>>>>>> I'm still not convinced around the fact that because the WALs > >>>>>>>>>> contain > >>>>>>>>>>>>> regions of multiple tables > >>>>>>>>>>>>> the incremental backup will keep around WALs with some data > >>>>>> that > >>>>>>>> we > >>>>>>>>>>> don't > >>>>>>>>>>>>> really want in the backup (for space or maybe security > >>>>>> reason). > >>>>>>>>>>>>> > >>>>>>>>>>>>> then there was the question about for how long should I take > >>>>>>>>>>>> incrementals, > >>>>>>>>>>>>> before deciding that a fresh full backup is less costly in > >>>>>> terms > >>>>>>>> of > >>>>>>>>>>>> space. > >>>>>>>>>>>>> but I think this incremental merge/compaction was a feature > on > >>>>>>> the > >>>>>>>>>>>> roadmap > >>>>>>>>>>>>> as Phase3. > >>>>>>>>>>>>> which I think is ok to get later on, > >>>>>>>>>>>>> maybe just call out a lifecycle example on the docs under > >>>>>> "best > >>>>>>>>>>>> practices". > >>>>>>>>>>>> > >>>>>>>>>>>> I think this will depend on the use case, and other factors > like > >>>>>>>>>>> bandwidth > >>>>>>>>>>>> available, how much data > >>>>>>>>>>>> the user is willing to lose in case of catastrophic failure > and > >>>>>>> how > >>>>>>>>>>>> "expensive" is full backup versus > >>>>>>>>>>>> incremental one. > >>>>>>>>>>>> > >>>>>>>>>>>> The full backup should also be useable by default, so maybe we > >>>>>> can > >>>>>>>>>> make > >>>>>>>>>>> an > >>>>>>>>>>>> option to not even keep WAL files, and completely disable > >>>>>>>> incremental > >>>>>>>>>>>> backups? > >>>>>>>>>>>> > >>>>>>>>>>>> Enis > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> has anyone interested in using backups looked at the doc in > >>>>>>>>>> HBASE-7912? > >>>>>>>>>>>>> is the current design of incremental backup acceptable for > >>>>>>>> everyone > >>>>>>>>>>>> wanting > >>>>>>>>>>>>> to use this feature? > >>>>>>>>>>>>> (maybe this should be a question for the @user list and not > >>>>>> dev) > >>>>>>>>>>>>> > >>>>>>>>>>>>> is there anyone already using this feature or it is just dev > >>>>>>>> testing > >>>>>>>>>>> it? > >>>>>>>>>>>>> to me will be interesting having a use-case/workflow example, > >>>>>>>>>>>>> to see if in the real world my concerns about incremental are > >>>>>>> not > >>>>>>>>>>> showing > >>>>>>>>>>>>> up. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Jul 19, 2016 at 1:35 PM, Ted Yu <yuzhih...@gmail.com > >>>>>>> <javascript:;>> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Gentle ping on this subject. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The changes are mostly non-intrusive. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> More comments are welcome. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 9:29 PM, Vladimir Rodionov < > >>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Not that hard, Andrew. I will open JIRA. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -Vlad > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 8:46 PM, Andrew Purtell < > >>>>>>>>>>>>>> andrew.purt...@gmail.com <javascript:;>> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> How hard would it be to convert what you've been using > >>>>>> to > >>>>>>>> test > >>>>>>>>>>> end > >>>>>>>>>>>> to > >>>>>>>>>>>>>> end > >>>>>>>>>>>>>>>> during dev into an IT? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Jul 11, 2016, at 5:31 PM, Vladimir Rodionov < > >>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If > >>>>>> not, > >>>>>>>> any > >>>>>>>>>>> tips > >>>>>>>>>>>>> on a > >>>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore > >>>>>>> them? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> We do not have yet, but we have a lot of unit tests. > >>>>>> We > >>>>>>>>>>> provide 2 > >>>>>>>>>>>>> API > >>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>> backup: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. Admin.getBackupAdmin > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2. Command - line via hbase command. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Everything is straightforward. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -Vlad > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 5:23 PM, Dima Spivak < > >>>>>>>>>>>>> dspi...@cloudera.com <javascript:;>> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Is there an integration test in hbase-it yet? If not, > >>>>>>> any > >>>>>>>>>> tips > >>>>>>>>>>>> on > >>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>> semi-automateable way to take backups and restore > >>>>>> them? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> -Dima > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 6:42 PM, Vladimir Rodionov < > >>>>>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Sorry, wrong links: > >>>>>>>>>>>>>>>>>>> These are the phases: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Phase 1: > >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- > >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14030 > >>>>>>>>> 14030 > >>>>>>>>>>>>>>>>>>> Phase 2: > >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- > >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14123 > >>>>>>>>> 14123 > >>>>>>>>>>>>>>>>>>> Phase 3: > >>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- > >>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-14414 > >>>>>>>>> 14414 > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> -Vlad > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 4:41 PM, Vladimir Rodionov < > >>>>>>>>>>>>>>>>>> vladrodio...@gmail.com <javascript:;> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> These are the phases: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Phase 1: > >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- > >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912 > >>>>>>>>> 14030 > >>>>>>>>>>>>>>>>>>>> Phase 2: > >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- > >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912 > >>>>>>>>> 14123 > >>>>>>>>>>>>>>>>>>>> Phase 3: > >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE- > >>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-7912 > >>>>>>>>> 14414 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> -Vlad > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Mon, Jul 11, 2016 at 12:21 PM, Enis Söztutar < > >>>>>>>>>>>>> e...@apache.org <javascript:;>> > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> As you guys may already be familiar, Vladimir, > >>>>>> Ted, > >>>>>>>>>> Jerry > >>>>>>>>>>> and > >>>>>>>>>>>>>>> others > >>>>>>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>>>> been developing the backup / restore functionality > >>>>>>> in > >>>>>>>> a > >>>>>>>>>>>> series > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> issues > >>>>>>>>>>>>>>>>>>>>> committed in the separate branch HBASE-7912[1]. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Backup / Restore functionality is tracked as a > >>>>>>> 4-phase > >>>>>>>>>>>> project, > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> first two phases are complete and useable. We are > >>>>>>> now > >>>>>>>>>>> working > >>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>> Phase 3 > >>>>>>>>>>>>>>>>>>>>> items, which are mostly improvements. We think > >>>>>> that > >>>>>>>> the > >>>>>>>>>>>> current > >>>>>>>>>>>>>>> code > >>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> branch containing all Phase 1 and Phase 2 items, > >>>>>> and > >>>>>>>>>> some > >>>>>>>>>>>>> Phase 3 > >>>>>>>>>>>>>>>>>> items > >>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>> useable on it's own, and we do not have to wait > >>>>>> for > >>>>>>>> all > >>>>>>>>>> the > >>>>>>>>>>>>>>>> subtickets > >>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> be finished to make it completely useable (as > >>>>>> follow > >>>>>>>> up > >>>>>>>>>>>> tickets > >>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>> mostly > >>>>>>>>>>>>>>>>>>>>> improvements or optimizations). The improvements > >>>>>> in > >>>>>>>> the > >>>>>>>>>>> works > >>>>>>>>>>>>> are > >>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>>> backwards compatible with the existing stuff. > >>>>>> Thus, > >>>>>>> we > >>>>>>>>>>> would > >>>>>>>>>>>>> like > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> propose that the branch HBASE-7912 be merged into > >>>>>>>>>> master. > >>>>>>>>>>>> The > >>>>>>>>>>>>>>> parent > >>>>>>>>>>>>>>>>>>> jira > >>>>>>>>>>>>>>>>>>>>> has a design doc that goes into details about the > >>>>>>>>>>>>> implementation > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>> design > >>>>>>>>>>>>>>>>>>>>> choices in case you are interested[2]. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Most of the changes are largely non-intrusive and > >>>>>>>>>> confined > >>>>>>>>>>> to > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> backup subsystem. > >>>>>>>>>>>>>>>>>>>>> The unit tests have been passing on manual runs > >>>>>> and > >>>>>>> we > >>>>>>>>>>>>>>> (hortonworks) > >>>>>>>>>>>>>>>>>>> have > >>>>>>>>>>>>>>>>>>>>> been running the integration tests as well as some > >>>>>>>> other > >>>>>>>>>>>>>>> shell-based > >>>>>>>>>>>>>>>>>>>>> system > >>>>>>>>>>>>>>>>>>>>> tests on a forked version of the code. Most of the > >>>>>>>> work > >>>>>>>>>> has > >>>>>>>>>>>>> been > >>>>>>>>>>>>>>>>>>> reviewed > >>>>>>>>>>>>>>>>>>>>> by 1, 2 or 3 committers already (mostly Ted, > >>>>>> myself > >>>>>>>> and > >>>>>>>>>>>> Jerry). > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> What do you guys think? Is it time to call a vote? > >>>>>>> Any > >>>>>>>>>>>> concerns > >>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>>>>> feedback > >>>>>>>>>>>>>>>>>>>>> appreciated. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> [1] > >>>>>>> https://issues.apache.org/jira/browse/HBASE-7912 > >>>>>>>>>>>>>>>>>>>>> [2] > >>>>>>> https://issues.apache.org/jira/secure/attachment/12816339/ > >>>>>> HBaseBackupAndRestore%20-0.91.pdf > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Enis > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> -Dima > >> >
