Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS) Refactoring work over in HBASE-16727 is ready for review. Kindly provide your feedback. Thanks On Mon, Oct 3, 2016 at 3:05 PM, Andrew Purtell <apurt...@apache.org> wrote: > This sounds good to me. &g

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

> obviously. > > > > ____________ > > From: Vladimir Rodionov <vladrodio...@gmail.com> > > Sent: Monday, September 26, 2016 11:48 AM > > To: dev@hbase.apache.org > > Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION]

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

odio...@gmail.com> > Sent: Monday, September 26, 2016 11:48 AM > To: dev@hbase.apache.org > Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs > started by Master or RS) > > Ok, we had internal discussion and this is what we are suggesting now: > > 1. We

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

ir Rodionov <vladrodio...@gmail.com> > Sent: Monday, September 26, 2016 11:48 AM > To: dev@hbase.apache.org > Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs > started by Master or RS) > > Ok, we had internal discussion and this is what we are suggesting now:

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

M To: dev@hbase.apache.org Subject: Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS) Ok, we had internal discussion and this is what we are suggesting now: 1. We will create separate module (hbase-backup) and move server-side code there. 2. Master and RS

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

;>>>>>>>>> :-) > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> So on this issue, we have these on the plate: > > >>>>>>>>>>

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

;>>>>>> external dependencies to HMaster, especially add more > >>>> works > >>>>>> for > >>>>>>>> the > >>>>>>>>>>> start > >>>>>>>>>>>>> up proces

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

; >>>>>>>>>> not > > > > > >>>>>>>>>>> -- > > > > > >>>>>>>>>>>> ever being able to launch MR jobs. > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> + MR is dead. W

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

on. > Lets > > > > >>>> not > > > > >>>>>>>> clutter > > > > >>>>>>>>>>> task > > > > >>>>>>>>>>>> harder by piling on more moving part

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

; > >>>>>>>>>>>>> Matteo > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu < > > > >>&g

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

gt;>>>>>> + Master is a rats nest of state. Matteo, Stephen, and Appy >>>>>>>> are >>>>>>>>>>> busy >>>>>>>>>>>>>>> working hard on moving it up on to a new foundation. Lets >>>

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

>>>>>>>>>> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu < >>>>>>> yuzhih...@gmail.com >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

t;>>>>>>> Master more stable. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Cheers > > >>>>>>>>>>>>>> > > >>>>>

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

>>> MR, >> >>>>>>>>>>>>>>>>> we >> >>>>>>>>>>>>>>>>>> can certainly consider that >> >>>>>>>>>>>>>>>>>> >> >>>&g

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

this time:) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Why I call the code ugly? Can you simply tell me the > >>>>>>> sequence > >>>>>>>>> of > &

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

; No, not your fault, at lease, not this time:) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Why I call the code ugly? Can you simply tell me the > >>>>>>> sequence > >>>>>

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

t;>>> compactions.. But I was thinking more about >>>> the >>>>>>>>>>> SpliceMachine >>>>>>>>>>>>>>>> approach >>>>>>>>>>>>>>>>> of >>>>

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

;>> for > >>>>>>>> the > >>>>>>>>>>> start > >>>>>>>>>>>>> up processing... > >>>>>>>>>>>>> > >>>>>>>>>>>>&

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

;>>>>>>>>> being used by HBase already for some >>> operations. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On (1), we have to deal with a myriad of >>> issues - >

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

> > > > problem if you do not store any metada locally. > But > > > the > > > > > > ugly > > > > > > > > code > > > > > > > > > > in > > > > > > > > > > > > > >

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

> > From: 张铎 <palomino...@gmail.com> > > > > > > > > > > > > > > > Sent: Thursday, September 22, 2016 8:32 PM > > > > > > > > > > > > > > > To: dev@hbase.apache.org

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

e... > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/HADOOP-13433 > > > > > > > > > > > > > > > > > > > > > > > > 2016-09-23 12:53 GMT+

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

; > > > > > > > >> If in the future, we find better ways of doing > this > > > > > without > > > > > > > > using > > > > > > > > > > MR, > > > > > > > > > > > we > > > > > > > > > > > &g

Re: Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

> > > > -Vlad > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das < > > > > > > d...@hortonworks.com > > > > > > > > > > > > > > > &g

Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

t; > > > > > managing compactions in Spark where apparently they > saw a > > > lot > > > > > of > > > > > > > > > > benefits. > > > > > > > > > > > Apologies for givin

Re: [DISCUSSION] MR jobs started by Master or RS

t; > > > > > > > > > compactions.. But I was thinking more about the > > > SpliceMachine > > > > > > > > approach > > > > > > > > > of > > > > > > > > > > > managing compactions in Spark where apparently they > s

Re: [DISCUSSION] MR jobs started by Master or RS

gt; > > > > > > > > > > > > > > So on this issue, we have these on the plate: > > > > > > > > > > 0. Somehow not use MR but something like that > > > > > > > > > > 1. Run a standalone service other than

Re: [DISCUSSION] MR jobs started by Master or RS

wer to (0), and I don't > think > > > > it's > > > > > > even > > > > > > > > > worth the effort of trying to build something when MR is > > > already > > > > > > there, > > > > > > > > and > &g

Re: [DISCUSSION] MR jobs started by Master or RS

> server > > > > > not > > > > > > > > being the least of them all. Security (kerberos > authentication, > > > > > another > > > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. > > &g

Re: [DISCUSSION] MR jobs started by Master or RS

e HBase Master. I haven't seen any > > good > > > > > reason > > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's > not > > > > > ideal; > > > > > > > agreed. > > > > > > >

Re: [DISCUSSION] MR jobs started by Master or RS

ber 23, 2016 8:40 AM To: dev Subject: Re: [DISCUSSION] MR jobs started by Master or RS That is the point, Matteo. Put it another way, there is nothing that stops a user from deploying custom procedure, custom co-processor that calls out MR job. The optional feature should satisfy some basic requir

Re: [DISCUSSION] MR jobs started by Master or RS

good > > > > > reason > > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's > not > > > > > ideal; > > > > > > > agreed. > > > > > > > > > > > > >

Re: [DISCUSSION] MR jobs started by Master or RS

t; > > > > > > substitute that (1) with the HBase Master. I haven't seen any > > good > > > > > reason > > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's > not > > > > > ideal; > > > &

Re: [DISCUSSION] MR jobs started by Master or RS

> > > > backup/restore jobs from the master. I think Ted has summarized > > some > > > of > > > > > the > > > > > > issues that we need to take care of - basically, the master can > > keep > > > > > track > > >

Re: [DISCUSSION] MR jobs started by Master or RS

d it fail, the backup master can continue > > > > keeping > > > > > track of it (since the jobId would have been recorded in the proc > > WAL). > > > > The > > > > > master can also do cleanup, etc. of failed backup/restore > processes. > &g

Re: [DISCUSSION] MR jobs started by Master or RS

hen we'd better implement it without > MR > > > > >>>>> dependency, like DLS. > > > > >>>>> > > > > >>>>> Thanks. > > > > >>>>> > > > > >>>>> 2016-09-23 10:1

Re: [DISCUSSION] MR jobs started by Master or RS

but I think the bottom line is that we > > should > > > >>>> launch > > > >>>>>> the jobs from outside manually or by other services. > > > >>>>>> > > > >>>>>> 2016-09-23 9:47 GM

Re: [DISCUSSION] MR jobs started by Master or RS

t using MR, we > can certainly consider that. But IMO don't think we should block this patch > from getting merged. > > ____ > From: 张铎 <palomino...@gmail.com> > Sent: Thursday, September 22, 2016 8:32 PM > To: dev@hbase.apache.org > Subj

Re: [DISCUSSION] MR jobs started by Master or RS

esign/arch point of view (maybe code review is still pending from Matteo). >> If in the future, we find better ways of doing this without using MR, we >> can certainly consider that. But IMO don't think we should block this patch >> from getting merged. >> >> >

Re: [DISCUSSION] MR jobs started by Master or RS

ng merged. > > ____________ > From: 张铎 <palomino...@gmail.com> > Sent: Thursday, September 22, 2016 8:32 PM > To: dev@hbase.apache.org > Subject: Re: [DISCUSSION] MR jobs started by Master or RS > > So what about a standalone service other than

Re: [DISCUSSION] MR jobs started by Master or RS

hout using MR, we > can certainly consider that. But IMO don't think we should block this patch > from getting merged. > > ____ > From: 张铎 <palomino...@gmail.com> > Sent: Thursday, September 22, 2016 8:32 PM > To: dev@hbase.apach

Re: [DISCUSSION] MR jobs started by Master or RS

ino...@gmail.com> Sent: Thursday, September 22, 2016 8:32 PM To: dev@hbase.apache.org Subject: Re: [DISCUSSION] MR jobs started by Master or RS So what about a standalone service other than master? You can use your own procedure store in that service? 2016-09-23 11:28 GMT+08:00 Ted Yu <yuzhih...@

Re: [DISCUSSION] MR jobs started by Master or RS

> >>>> launch > > > >>>>>> the jobs from outside manually or by other services. > > > >>>>>> > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell < > > andrew.purt...@gmail.com > > > >: > > > >>>>>> > > > >>&g

Re: [DISCUSSION] MR jobs started by Master or RS

gt;>> > > >>>>>>> Can this be driven by a utility derived from Tool like our other > MR > > >>>> apps? > > >>>>>>> The issue is needing the AccessController to decide if allowed? > But > > >>>

Re: [DISCUSSION] MR jobs started by Master or RS

>>> > >>>>>>> Can this be driven by a utility derived from Tool like our other MR > >>>> apps? > >>>>>>> The issue is needing the AccessController to decide if allowed? But > >>>> nothing > >>&

Re: [DISCUSSION] MR jobs started by Master or RS

ents the user from running the job manually/independently, >>> right? >>>>>>>> >>>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi < >>>>> theo.berto...@gmail.com> >>>>>>>> wrote: >>&g

Re: [DISCUSSION] MR jobs started by Master or RS

t;>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi < >>> theo.berto...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> just a remark. my query was not about tools using MR (everyone i >>> think >>>

Re: [DISCUSSION] MR jobs started by Master or RS

dently, >>> right? >>> >> >>> >>> >> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi < >>> >> theo.berto...@gmail.com> >>> >> >>> wrote: >>> >> >>> > >>>

Re: [DISCUSSION] MR jobs started by Master or RS

gt;> >> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi < >> >> theo.berto...@gmail.com> >> >> >>> wrote: >> >> >>> > >> >> >>> > just a remark. my query was not about tools using MR (everyone i

Re: [DISCUSSION] MR jobs started by Master or RS

nce this will be the first time we do this > > >>> > > > >>> > Matteo > > >>> > > > >>> > > > >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das < > d...@hortonworks.com> > > >&

Re: [DISCUSSION] MR jobs started by Master or RS

jobs from Master and > RSs > >>> > code?" since this will be the first time we do this > >>> > > >>> > Matteo > >>> > > >>> > > >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <d...@hortonwor

Re: [DISCUSSION] MR jobs started by Master or RS

gt; > >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <d...@hortonworks.com> >>> wrote: >>> >> >>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's >>> >> fine to be dependent on MR. MR is the right framework for such.

Re: [DISCUSSION] MR jobs started by Master or RS

tSnapshot / Backup / Restore, it's > >> fine to be dependent on MR. MR is the right framework for such. We > should > >> also do compactions using MR (just saying :) ) > >> ____________ > >> From: Ted Yu <yuzhih...@gmail.com>

Re: [DISCUSSION] MR jobs started by Master or RS

; >> Sent: Thursday, September 22, 2016 2:00 PM >> To: dev@hbase.apache.org >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS >> >> I agree - backup / restore is in the same category as import / export. >> >> On Thu, Sep 22, 2016 at 1:58 PM,

Re: [DISCUSSION] MR jobs started by Master or RS

s the right framework for such. We should also do > compactions using MR (just saying :) ) > > From: Ted Yu <yuzhih...@gmail.com> > Sent: Thursday, September 22, 2016 2:00 PM > To: dev@hbase.apache.org > Subject: Re: [DISCUSSION] MR jobs

Re: [DISCUSSION] MR jobs started by Master or RS

R. MR is the right framework for such. We should >> also do compactions using MR (just saying :) ) >> >> From: Ted Yu <yuzhih...@gmail.com> >> Sent: Thursday, September 22, 2016 2:00 PM >> To: dev@hbase.apache.org >&g

Re: [DISCUSSION] MR jobs started by Master or RS

From: Jean-Marc Spaggiari <jean-m...@spaggiari.org> > Sent: Thursday, September 22, 2016 4:29 PM > To: dev > Subject: Re: [DISCUSSION] MR jobs started by Master or RS > > Well, I'm just not using those features ;) But was hopping for the MOBs ;) > My point is, if we can do it w

Re: [DISCUSSION] MR jobs started by Master or RS

R? Curious. From: Jean-Marc Spaggiari <jean-m...@spaggiari.org> Sent: Thursday, September 22, 2016 4:29 PM To: dev Subject: Re: [DISCUSSION] MR jobs started by Master or RS Well, I'm just not using those features ;) But was hopping for the MOBs ;) My point is, if we can

Re: [DISCUSSION] MR jobs started by Master or RS

<theo.berto...@gmail.com> Sent: Thursday, September 22, 2016 3:44 PM To: dev@hbase.apache.org Subject: Re: [DISCUSSION] MR jobs started by Master or RS just a remark. my query was not about tools using MR (everyone i think is ok with those). the topic was about: "are we ok with running M

Re: [DISCUSSION] MR jobs started by Master or RS

Devaraj Das <d...@hortonworks.com> > >> wrote: > >> > > >> > > Very much agree; for tools like ExportSnapshot / Backup / Restore, > >> it's > >> > > fine to be dependent on MR. MR is the right framework for such. We > >> sh

Re: [DISCUSSION] MR jobs started by Master or RS

> > >> > >> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <d...@hortonworks.com> >> wrote: >> > >> > > Very much agree; for tools like ExportSnapshot / Backup / Restore, >> it's >> > > fine to be dependent on MR. MR is the right fr

Re: [DISCUSSION] MR jobs started by Master or RS

:) ) > > > ________________ > > > From: Ted Yu <yuzhih...@gmail.com> > > > Sent: Thursday, September 22, 2016 2:00 PM > > > To: dev@hbase.apache.org > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS > > >

Re: [DISCUSSION] MR jobs started by Master or RS

> > fine to be dependent on MR. MR is the right framework for such. We should > > also do compactions using MR (just saying :) ) > > > > From: Ted Yu <yuzhih...@gmail.com> > > Sent: Thursday, September 22, 2016 2:00 PM > &

Re: [DISCUSSION] MR jobs started by Master or RS

hih...@gmail.com> > Sent: Thursday, September 22, 2016 2:00 PM > To: dev@hbase.apache.org > Subject: Re: [DISCUSSION] MR jobs started by Master or RS > > I agree - backup / restore is in the same category as import / export. > > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell

Re: [DISCUSSION] MR jobs started by Master or RS

tember 22, 2016 2:00 PM To: dev@hbase.apache.org Subject: Re: [DISCUSSION] MR jobs started by Master or RS I agree - backup / restore is in the same category as import / export. On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <andrew.purt...@gmail.com> wrote: > Backup is extra tooling ar

Re: [DISCUSSION] MR jobs started by Master or RS

I agree - backup / restore is in the same category as import / export. On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell wrote: > Backup is extra tooling around core in my opinion. Like import or export. > Or the optional MOB tool. It's fine. > > > On Sep 22, 2016, at

Re: [DISCUSSION] MR jobs started by Master or RS

Backup is extra tooling around core in my opinion. Like import or export. Or the optional MOB tool. It's fine. > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi wrote: > > What's the latest opinion around running MR jobs from hbase (Master or RS)? > > I remember in the

Re: [DISCUSSION] MR jobs started by Master or RS

I would be -1 a requirement for MR for something core to HBase. > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi wrote: > > What's the latest opinion around running MR jobs from hbase (Master or RS)? > > I remember in the past that there was discussion about not having MR

[DISCUSSION] MR jobs started by Master or RS

What's the latest opinion around running MR jobs from hbase (Master or RS)? I remember in the past that there was discussion about not having MR has direct dependency of hbase. I think some of discussion where around MOB that had a MR job to compact, that later was transformed in a non-MR job to